perf: Check if HT is supported and enabled

Avoid the percore allocations if HT is not supported or disabled.

Signed-off-by Lin Ming <[email protected]>

---
 arch/x86/kernel/cpu/perf_event_intel.c |   44 +++++++++++++++++++++++++++----
 1 files changed, 38 insertions(+), 6 deletions(-)

arch/x86/kernel/cpu/perf_event_intel.c +16 -0

index bc4afb1..354d1de 100644
@@ -1053,9 +1053,38 @@ static __initconst const struct x86_pmu core_pmu = {
       .event_constraints      = intel_core_event_constraints,
 };
 
+/*
+ * Check if HT is capable and enabled
+ * TBD: move it to generic place, so it can be used by others
+ */
+
+static bool ht_enabled(int cpu)
+{
+       int total_logical_processors, total_cores;
+       unsigned int eax, ebx, ecx, edx;
+
+       cpuid(1, &eax, &ebx, &ecx, &edx);
+
+       /* Bit 28 in EDX indicates if it's HT capable */
+       if (!(edx & 0x10000000))
+               return false;
+

arch/x86/kernel/cpu/perf_event_intel.c +12 -7

 static int intel_pmu_cpu_prepare(int cpu)
 {
       struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
+
+       if (!ht_enabled(cpu))
+               return NOTIFY_OK;
 
       cpuc->per_core = kzalloc_node(sizeof(struct intel_percore),
                                     GFP_KERNEL, cpu_to_node(cpu));

@@ -1073,6 +1102,15 @@ static void intel_pmu_cpu_starting(int cpu)
       int core_id = topology_core_id(cpu);
       int i;
 
+       init_debug_store_on_cpu(cpu);
+       /*
+        * Deal with CPUs that don't clear their LBRs on power-up.
+        */
+       intel_pmu_lbr_reset();
+
+       if (!ht_enabled(cpu))
+               return;
+
       for_each_online_cpu(i) {
               struct intel_percore *pc = per_cpu(cpu_hw_events, i).per_core;
 

@@ -1085,12 +1123,6 @@ static void intel_pmu_cpu_starting(int cpu)
 
       cpuc->per_core->core_id = core_id;
       cpuc->per_core->refcnt++;
-
-       init_debug_store_on_cpu(cpu);
-       /*
-        * Deal with CPUs that don't clear their LBRs on power-up.
-        */
-       intel_pmu_lbr_reset();
 }
 
 static void intel_pmu_cpu_dying(int cpu)
--

1.7.3

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 11 lines

arch/x86/kernel/cpu/perf_event_intel.c +11 -0

+static bool ht_enabled(int cpu)
+{
+       int total_logical_processors, total_cores;
+       unsigned int eax, ebx, ecx, edx;
+
+       cpuid(1, &eax, &ebx, &ecx, &edx);
+
+       /* Bit 28 in EDX indicates if it's HT capable */
+       if (!(edx & 0x10000000))
+               return false;
+

arch/x86/kernel/cpu/perf_event_intel.c static __initconst const struct x86_pmu core_pmu = { · line 1081

Quoted 10 lines

arch/x86/kernel/cpu/perf_event_intel.c +10 -0

+       total_logical_processors = (ebx >> 16) & 0xff;
+
+       ecx = 0;
+       cpuid(4, &eax, &ebx, &ecx, &edx);
+       total_cores = ((eax >> 26) & 0x3f) + 1;
+
+       /* Thread nums per core */
+       return (total_logical_processors / total_cores) > 1;
+}
+

This function can be simplified as below,

static bool ht_enabled()

{
        if (!cpu_has(&boot_cpu_data, X86_FEATURE_HT))
                return false;

        return smp_num_siblings > 1;
}

But this still can't detect if HT is on or off. smp_num_siblings is always 2 even if HT is disabled in BIOS.

Any idea how to detect if HT is on or not?

Not quite sure, the intel docs aren't really clear on how the HW supports HT, has 2 siblings but BIOS disabled it thing works. I just tried reading the arch/x86 code but that only got me more confused.

Quoted 14 lines

This function can be simplified as below,

static bool ht_enabled()

{
        if (!cpu_has(&boot_cpu_data, X86_FEATURE_HT))
                return false;

        return smp_num_siblings > 1;
}

But this still can't detect if HT is on or off. smp_num_siblings is always 2 even if HT is disabled in BIOS.

Quoted 1 line

Any idea how to detect if HT is on or not?

Not quite sure, the intel docs aren't really clear on how the HW supports HT, has 2 siblings but BIOS disabled it thing works. I just tried reading the arch/x86 code but that only got me more confused.

hpa, could you comment?

Quoted 14 lines · Peter Zijlstra

Quoted 14 lines

This function can be simplified as below,

static bool ht_enabled()

{
        if (!cpu_has(&boot_cpu_data, X86_FEATURE_HT))
                return false;

        return smp_num_siblings > 1;
}

But this still can't detect if HT is on or off. smp_num_siblings is always 2 even if HT is disabled in BIOS.

Quoted 8 lines

Quoted 1 line

Any idea how to detect if HT is on or not?

Not quite sure, the intel docs aren't really clear on how the HW supports HT, has 2 siblings but BIOS disabled it thing works. I just tried reading the arch/x86 code but that only got me more confused.

hpa, could you comment?

Zeroeth of all: anyone who writes () in a function prototype in C needs to get severely napalmed, maimed, hung and then really hurt. It is (void) in C, () means (...) which is literally NEVER what you want.

Now, *first* of all: smp_num_siblings as it sits is obviously broken; the whole notion of a global variable for what is inherently a per-cpu property is just braindead. At least theoretically there could be cores with and without HT or with different thread count in the same system.

Second, perf should get this from /proc/cpuinfo, not by grotting around cpuid by itself.

Third, the code in the kernel is indeed pretty confusing, and it also has the global variable braindamage... but either it works (in which case getting the data from /proc/cpuinfo works) or it needs fixing in addition to the global variable issue.

Quoted 28 lines · H. Peter Anvin

Quoted 14 lines · Peter Zijlstra

Quoted 14 lines

This function can be simplified as below,

static bool ht_enabled()
{
        if (!cpu_has(&boot_cpu_data, X86_FEATURE_HT)) return false;

        return smp_num_siblings > 1;
}

But this still can't detect if HT is on or off. smp_num_siblings is always 2 even if HT is disabled in BIOS.

Quoted 8 lines

Quoted 1 line

Any idea how to detect if HT is on or not?

Not quite sure, the intel docs aren't really clear on how the HW supports HT, has 2 siblings but BIOS disabled it thing works. I just tried reading the arch/x86 code but that only got me more confused.

hpa, could you comment?

Zeroeth of all: anyone who writes () in a function prototype in C needs to get severely napalmed, maimed, hung and then really hurt.  It is (void) in C, () means (...) which is literally NEVER what you want.

Quoted 7 lines

Now, *first* of all: smp_num_siblings as it sits is obviously broken; the whole notion of a global variable for what is inherently a per-cpu property is just braindead.  At least theoretically there could be cores with and without HT or with different thread count in the same system.

Second, perf should get this from /proc/cpuinfo, not by grotting around cpuid by itself.

Its the kernel space bit that wants to know if HT is enabled on CPU bringup, /proc/cpuinfo is kinda useless in that context.

Quoted 4 lines

Third, the code in the kernel is indeed pretty confusing, and it also has the global variable braindamage... but either it works (in which case getting the data from /proc/cpuinfo works) or it needs fixing in addition to the global variable issue.

So you failed to address the primary question, how do we tell if HT is enabled for a particular CPU?

X86_FEATURE_HT tells us the CPU supports telling us about HT, smp_num_siblings > 1 tells us the CPU is capable of HT. But Lin found that if you disable HT in the BIOS both those are true but we still don't have HT enabled.

The problem we have to address is that Intel has some MSRs shared between threads and if HT is enabled we need to allocate some memory to manage this shared resource.

The simple check outlined above X86_FEATURE_HT && smp_num_siblings > 1, will get us most of the way and will I think be good enough (false positives but no false negatives). But it did get us wondering about how all this works.

So could you, irrespective of Linux implementation details tell us how to detect if HT is available and enabled from a HW and BIOS perspective?

My solution at the time (2.6.30) was to do:

        ht_enabled = cpumask_weight(__get_cpu_var(cpu_sibling_map)) > 1;

Quoted 2 lines · Stephane Eranian

My solution at the time (2.6.30) was to do:
        ht_enabled = cpumask_weight(__get_cpu_var(cpu_sibling_map)) > 1;

Won't that report a machine a HT disabled when you offline a sibling?

Which kinda defeats the purpose of our usage here, since we need to know it before either sibling comes online.

Couldn't you simply over-provision, and then when the CPU is online, use my ht_enabled statement to figure out whether or not you need to handle the sharing issue?

Quoted 4 lines · Stephane Eranian

Couldn't you simply over-provision, and then when the CPU is online, use my ht_enabled statement to figure out whether or not you need to handle the sharing issue?

That's pretty much what Lin's latest does:

 X86_FEATURE_HT && smp_num_siblings > 1

shows the CPU is capable of HT and will allocate the needed resources.

X86_FEATURE_HT tells us the CPU supports telling us about HT, smp_num_siblings > 1 tells us the CPU is capable of HT.

Quoted 4 lines · Stephane Eranian

Couldn't you simply over-provision, and then when the CPU is online, use my ht_enabled statement to figure out whether or not you need to handle the sharing issue?

That's pretty much what Lin's latest does:

X86_FEATURE_HT && smp_num_siblings > 1

shows the CPU is capable of HT and will allocate the needed resources.

From the sound of it, it doesn't seem like it's worth optimizing out the

Quoted 5 lines · Peter Zijlstra

Quoted 2 lines · Stephane Eranian

My solution at the time (2.6.30) was to do:

        ht_enabled = cpumask_weight(__get_cpu_var(cpu_sibling_map)) > 1;

Won't that report a machine a HT disabled when you offline a sibling?

I think you're right. I was not dealing with hotplug CPU.

Quoted 2 lines

Which kinda defeats the purpose of our usage here, since we need to know it before either sibling comes online.

Then, it seems the only hope is to peek at a MSR that reports the BIOS setting. But I don't know which one it is.

Couldn't you simply over-provision, and then when the CPU is online, use my ht_enabled statement to figure out whether or not you need to handle the sharing issue?

If this is worth doing, then perhaps you could write a routine to check if another HT thread on this same core has already been brought online ... i.e. defer the allocation to when you see the second thread. If HT is off, then you'll never see a second thread so you'll avoid the allocation

static void intel_pmu_cpu_starting(int cpu)

{
        struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);
        struct intel_percore *pc;
        int core_id = topology_core_id(cpu);
        int i;

        init_debug_store_on_cpu(cpu);
        /*
         * Deal with CPUs that don't clear their LBRs on power-up.

         */
        intel_pmu_lbr_reset();

        if (!ht_enabled(cpu))
                return;

        cpuc->per_core = kzalloc_node(sizeof(struct intel_percore),
                                      GFP_KERNEL, cpu_to_node(cpu));

        if (!cpuc->per_core)
                return;

        raw_spin_lock_init(&cpuc->per_core->lock);
        pc = cpuc->per_core;

        for_each_cpu(i, topology_thread_cpumask(cpu)) {
                cpuc = &per_cpu(cpu_hw_events, cpu);

                cpuc->per_core = pc;
                cpuc->per_core->core_id = core_id;
                cpuc->per_core->refcnt++;
        }
}