Skip to contentSkip to conversation

Fix panic in logical packages calculation

PATCH In review Prarit Bhargava 4 participants 16 messages Nov 14, 2017 – Feb 7, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v6 cover Fix panic in logical packages calculation

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

Andi noted that the logical_packages calculation wastes 128k on 64-bit x86 systems and posted patches to fix the allocation. His patches result in significant memory size savings, and I have put my fix on top of those patches.

Here's an example of the panic:

     smpboot: Booting Node   1, Processors  #1 OK
     smpboot: Package 1 of CPU 1 exceeds BIOS package data 1.
     ------------[ cut here ]------------
     kernel BUG at arch/x86/kernel/cpu/common.c:1087!
     invalid opcode: 0000 [#1] SMP
     Modules linked in:
     CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.0-rc2+ #4

This was tested by dropping the number of active cores across all packages to reproduce the bug above. Additional testing included 2 socket and 4 socket systems and hotplugging entire sockets in different order.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>

Andi Kleen (2):
perf/x86/intel/uncore: Cache logical pkg id in uncore driver x86/topology: Avoid wasting 128k for package id array

Prarit Bhargava (1):
x86/smpboot: Fix __max_logical_packages estimate

 arch/x86/events/intel/uncore.c       |   4 +-
 arch/x86/events/intel/uncore.h       |   2 +-
 arch/x86/events/intel/uncore_snbep.c |   2 +-
 arch/x86/include/asm/processor.h     |   1 +
 arch/x86/kernel/smpboot.c            | 130 +++++++++++------------------------
 5 files changed, 44 insertions(+), 95 deletions(-)
Signature · 1 line
2.15.0.rc0.39.g2f0e14e64
#1 lore raw

PATCH v6 1/3 perf/x86/intel/uncore: Cache logical pkg id in uncore driver

From: Andi Kleen <[email protected]>

The SNB-EP uncore driver is the only user of topology_phys_to_logical_pkg in a performance critical path. Change it query the logical pkg ID only once at initialization time and then cache it in box structure.

[v2]: Used the existing pkgid, Logical package ID, which was already stored.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/events/intel/uncore.c       | 4 ++--
 arch/x86/events/intel/uncore.h       | 2 +-
 arch/x86/events/intel/uncore_snbep.c | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/events/intel/uncore.c +2 -2
index d45e06346f14..7874c980d569 100644
@@ -975,10 +975,10 @@ static void uncore_pci_remove(struct pci_dev *pdev)
 	int i, phys_id, pkg;
 
 	phys_id = uncore_pcibus_to_physid(pdev->bus);
-	pkg = topology_phys_to_logical_pkg(phys_id);
 
 	box = pci_get_drvdata(pdev);
 	if (!box) {
+		pkg = topology_phys_to_logical_pkg(phys_id);
 		for (i = 0; i < UNCORE_EXTRA_PCI_DEV_MAX; i++) {
 			if (uncore_extra_pci_dev[pkg].dev[i] == pdev) {
 				uncore_extra_pci_dev[pkg].dev[i] = NULL;
@@ -994,7 +994,7 @@ static void uncore_pci_remove(struct pci_dev *pdev)
 		return;
 
 	pci_set_drvdata(pdev, NULL);
-	pmu->boxes[pkg] = NULL;
+	pmu->boxes[box->pkgid] = NULL;
 	if (atomic_dec_return(&pmu->activeboxes) == 0)
 		uncore_pmu_unregister(pmu);
 	uncore_box_exit(box);
arch/x86/events/intel/uncore.h +1 -1
index 4364191e7c6b..414dc7e7c950 100644
@@ -100,7 +100,7 @@ struct intel_uncore_extra_reg {
 
 struct intel_uncore_box {
 	int pci_phys_id;
-	int pkgid;
+	int pkgid;	/* Logical package ID */
 	int n_active;	/* number of active events */
 	int n_events;
 	int cpu;	/* cpu to collect events */
arch/x86/events/intel/uncore_snbep.c +1 -1
index 95cb19f4e06f..de8f8625213c 100644
@@ -1057,7 +1057,7 @@ static void snbep_qpi_enable_event(struct intel_uncore_box *box, struct perf_eve
 
 	if (reg1->idx != EXTRA_REG_NONE) {
 		int idx = box->pmu->pmu_idx + SNBEP_PCI_QPI_PORT0_FILTER;
-		int pkg = topology_phys_to_logical_pkg(box->pci_phys_id);
+		int pkg = box->pkgid;
 		struct pci_dev *filter_pdev = uncore_extra_pci_dev[pkg].dev[idx];
 
 		if (filter_pdev) {
Signature · 1 line
2.15.0.rc0.39.g2f0e14e64
#2 lore raw

[tip:x86/urgent] perf/x86/intel/uncore: Cache logical pkg id in uncore driver

perf/x86/intel/uncore: Cache logical pkg id in uncore driver
Commit-ID:  d46b4c1ce5f0d9a13fb2318763076442669a2bdc
Gitweb:     https://git.kernel.org/tip/d46b4c1ce5f0d9a13fb2318763076442669a2bdc
Author:     Andi Kleen <[email protected]>
AuthorDate: Tue, 14 Nov 2017 07:42:55 -0500
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 17 Nov 2017 16:22:30 +0100

perf/x86/intel/uncore: Cache logical pkg id in uncore driver

The SNB-EP uncore driver is the only user of topology_phys_to_logical_pkg in a performance critical path.

Change it query the logical pkg ID only once at initialization time and then cache it in box structure. This allows to change the logical package management without affecting the performance critical path.

Cc Tom Lendacky <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kan Liang <[email protected]>
Cc He Chen <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tim Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
---
 arch/x86/events/intel/uncore.c       | 4 ++--
 arch/x86/events/intel/uncore.h       | 2 +-
 arch/x86/events/intel/uncore_snbep.c | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
arch/x86/events/intel/uncore.c +2 -2
index d45e063..7874c98 100644
@@ -975,10 +975,10 @@ static void uncore_pci_remove(struct pci_dev *pdev)
 	int i, phys_id, pkg;
 
 	phys_id = uncore_pcibus_to_physid(pdev->bus);
-	pkg = topology_phys_to_logical_pkg(phys_id);
 
 	box = pci_get_drvdata(pdev);
 	if (!box) {
+		pkg = topology_phys_to_logical_pkg(phys_id);
 		for (i = 0; i < UNCORE_EXTRA_PCI_DEV_MAX; i++) {
 			if (uncore_extra_pci_dev[pkg].dev[i] == pdev) {
 				uncore_extra_pci_dev[pkg].dev[i] = NULL;
@@ -994,7 +994,7 @@ static void uncore_pci_remove(struct pci_dev *pdev)
 		return;
 
 	pci_set_drvdata(pdev, NULL);
-	pmu->boxes[pkg] = NULL;
+	pmu->boxes[box->pkgid] = NULL;
 	if (atomic_dec_return(&pmu->activeboxes) == 0)
 		uncore_pmu_unregister(pmu);
 	uncore_box_exit(box);
arch/x86/events/intel/uncore.h +1 -1
index 4364191..414dc7e 100644
@@ -100,7 +100,7 @@ struct intel_uncore_extra_reg {
 
 struct intel_uncore_box {
 	int pci_phys_id;
-	int pkgid;
+	int pkgid;	/* Logical package ID */
 	int n_active;	/* number of active events */
 	int n_events;
 	int cpu;	/* cpu to collect events */
arch/x86/events/intel/uncore_snbep.c +1 -1
index 95cb19f..de8f862 100644
@@ -1057,7 +1057,7 @@ static void snbep_qpi_enable_event(struct intel_uncore_box *box, struct perf_eve
 
 	if (reg1->idx != EXTRA_REG_NONE) {
 		int idx = box->pmu->pmu_idx + SNBEP_PCI_QPI_PORT0_FILTER;
-		int pkg = topology_phys_to_logical_pkg(box->pci_phys_id);
+		int pkg = box->pkgid;
 		struct pci_dev *filter_pdev = uncore_extra_pci_dev[pkg].dev[idx];
 
 		if (filter_pdev) {
#3 lore raw

PATCH v6 2/3 x86/topology: Avoid wasting 128k for package id array

In reply to Prarit Bhargava

From: Andi Kleen <[email protected]>

I was looking at large early boot allocations and noticed that since (1f12e32f x86/topology: Create logical package id) every 64bit system allocates a 128k array to convert logical package ids.

This happens because the array is sized for MAX_LOCAL_APIC and that is always 32k on 64bit systems, and it needs 4 bytes for each entry.

This is fairly wasteful, especially for the common case of having only one socket, where we need 128K just to store a single 4 byte value.

Store the logical processor id in cpudata. Add a set flag that indicates that the cpu's cpudata entry has been written.

[v2]: Decrease logical_packages when the last thread in a socket is removed.
[v3]: Add more logic to keep logical and physical package IDs in synch.
[v4]: Keep logical mapping static by using hybrid approach of a small logical
to physical array and keeping logical cpu information in cpu_data. [v5]: Change kmalloc to GFP_ATOMIC to fix "sleeping function" warning on virtual machines. Remove phys_pkg_id. Add spinlock to avoid concurrency issues.
[v6]: Revert back to storing logical processor ID in cpudata, and add set flag to indicate that boot_cpu_data was written to a cpu's cpu_data.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/include/asm/processor.h |  1 +
 arch/x86/kernel/smpboot.c        | 75 ++++++++++++++++------------------------
 2 files changed, 30 insertions(+), 46 deletions(-)
arch/x86/include/asm/processor.h +1 -0
index bdac19ab2488..44fd3512442f 100644
@@ -132,6 +132,7 @@ struct cpuinfo_x86 {
 	/* Index into per_cpu list: */
 	u16			cpu_index;
 	u32			microcode;
+	unsigned		set : 1;
 } __randomize_layout;
 
 struct cpuid_regs {
arch/x86/kernel/smpboot.c +29 -46

133 lines not on this page · raw message

Signature · 1 line
2.15.0.rc0.39.g2f0e14e64
#4 lore raw

Re: [PATCH v6 2/3] x86/topology: Avoid wasting 128k for package id array

arch/x86/kernel/smpboot.c void __init smp_store_boot_cpu_info(void) · line 372

Quoted 5 lines · Prarit Bhargava
arch/x86/kernel/smpboot.c +1 -0
@@ -389,6 +369,7 @@ void __init smp_store_boot_cpu_info(void)
 	*c = boot_cpu_data;
 	c->cpu_index = id;
 	smp_init_package_map(c, id);
+	cpu_data(id).set = 1;

What's wrong with

       c->set ?

Aside of that 'set' is not really descriptive. initialized might be better suited.

Quoted 7 lines
arch/x86/kernel/smpboot.c +0 -0
 }
 
 /*
@@ -399,13 +380,15 @@ void smp_store_cpu_info(int id)
 {
 	struct cpuinfo_x86 *c = &cpu_data(id);
 

arch/x86/kernel/smpboot.c void smp_store_cpu_info(int id) · line 391

Quoted 10 lines
arch/x86/kernel/smpboot.c +3 -1
-	*c = boot_cpu_data;
+	if (cpu_data(id).set == 0)
+		*c = boot_cpu_data;
 	c->cpu_index = id;
 	/*
 	 * During boot time, CPU0 has this setup already. Save the info when
 	 * bringing up AP or offlined CPU0.
 	 */
 	identify_secondary_cpu(c);
+	cpu_data(id).set = 1;

See above.

No need to resend. I'll fix it up.

Thanks,

tglx

#5 lore raw

[tip:x86/urgent] x86/topology: Avoid wasting 128k for package id array

In reply to Prarit Bhargava
x86/topology: Avoid wasting 128k for package id array
Commit-ID:  30bb9811856f667042e746d8033883b1091a46ce
Gitweb:     https://git.kernel.org/tip/30bb9811856f667042e746d8033883b1091a46ce
Author:     Andi Kleen <[email protected]>
AuthorDate: Tue, 14 Nov 2017 07:42:56 -0500
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 17 Nov 2017 16:22:30 +0100

x86/topology: Avoid wasting 128k for package id array

Analyzing large early boot allocations unveiled the logical package id storage as a prominent memory waste. Since commit 1f12e32f4cd5 ("x86/topology: Create logical package id") every 64-bit system allocates a 128k array to convert logical package ids.

This happens because the array is sized for MAX_LOCAL_APIC which is always 32k on 64bit systems, and it needs 4 bytes for each entry.

This is fairly wasteful, especially for the common case of having only one socket, which uses exactly 4 byte out of 128K.

There is no user of the package id map which is performance critical, so the lookup is not required to be O(1). Store the logical processor id in cpu_data and use a loop based lookup.

To keep the mapping stable accross cpu hotplug operations, add a flag to cpu_data which is set when the CPU is brought up the first time. When the flag is set, then cpu_data is not reinitialized by copying boot_cpu_data on subsequent bringups.

[ tglx: Rename the flag to 'initialized', use proper pointers instead of
repeated cpu_data(x) evaluation and massage changelog. ]

Cc Tom Lendacky <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kan Liang <[email protected]>
Cc He Chen <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tim Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
---
 arch/x86/include/asm/processor.h |  1 +
 arch/x86/kernel/smpboot.c        | 73 ++++++++++++++++------------------------
 2 files changed, 30 insertions(+), 44 deletions(-)
arch/x86/include/asm/processor.h +1 -0
index 2db7cf7..cc16fa8 100644
@@ -132,6 +132,7 @@ struct cpuinfo_x86 {
 	/* Index into per_cpu list: */
 	u16			cpu_index;
 	u32			microcode;
+	unsigned		initialized : 1;
 } __randomize_layout;
 
 struct cpuid_regs {
arch/x86/kernel/smpboot.c +29 -44

129 lines not on this page · raw message

#6 lore raw

PATCH v6 3/3 x86/smpboot: Fix __max_logical_packages estimate

In reply to Prarit Bhargava

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

ie) On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +10 -45
index 838d36ff7ba6..2e3c5a394e79 100644
@@ -308,12 +308,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg)
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -323,44 +317,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -368,7 +324,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	cpu_data(id).set = 1;
 }
 
@@ -1371,7 +1327,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);
Signature · 1 line
2.15.0.rc0.39.g2f0e14e64
#7 lore raw

[tip:x86/urgent] x86/smpboot: Fix __max_logical_packages estimate

x86/smpboot: Fix __max_logical_packages estimate
Commit-ID:  b4c0a7326f5dc0ef7a64128b0ae7d081f4b2cbd1
Gitweb:     https://git.kernel.org/tip/b4c0a7326f5dc0ef7a64128b0ae7d081f4b2cbd1
Author:     Prarit Bhargava <[email protected]>
AuthorDate: Tue, 14 Nov 2017 07:42:57 -0500
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 17 Nov 2017 16:22:31 +0100

x86/smpboot: Fix __max_logical_packages estimate

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low.

This occurs when the total number of active cores across all packages is less than the maximum core count for a single package. e.g.:

On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Tom Lendacky <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Kan Liang <[email protected]>
Cc He Chen <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Tim Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +10 -45
index da5e162..3d01df7 100644
@@ -310,12 +310,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg) {
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -326,44 +320,6 @@ found:
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -371,7 +327,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	c->initialized = true;
 }
 
@@ -1341,7 +1297,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);
#8 lore raw

Re: [v6, 3/3] x86/smpboot: Fix __max_logical_packages estimate

In reply to Prarit Bhargava

Prarit Bhargava:

Quoted 116 lines

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

ie) On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +3 -45
index 838d36ff7ba6..2e3c5a394e79 100644
@@ -308,12 +308,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg)
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -323,44 +317,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -368,7 +324,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	cpu_data(id).set = 1;
 }
 
@@ -1371,7 +1327,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");

arch/x86/kernel/smpboot.c void __init native_smp_prepare_boot_cpu(void) · line 1342

Quoted 10 lines
arch/x86/kernel/smpboot.c +7 -0
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

Re: [v6,3/3] x86/smpboot: Fix __max_logical_packages estimate

In reply to Prarit Bhargava

Prarit Bhargava:

Quoted 116 lines

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

ie) On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +3 -45
index 838d36ff7ba6..2e3c5a394e79 100644
@@ -308,12 +308,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg)
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -323,44 +317,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -368,7 +324,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	cpu_data(id).set = 1;
 }
 
@@ -1371,7 +1327,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");

arch/x86/kernel/smpboot.c void __init native_smp_prepare_boot_cpu(void) · line 1342

Quoted 10 lines
arch/x86/kernel/smpboot.c +7 -0
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

#10 lore raw

Re: [v6, 3/3] x86/smpboot: Fix __max_logical_packages estimate

Quoted 125 lines · Simon Gaiser

Prarit Bhargava:

Quoted 124 lines

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

ie) On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +10 -45
index 838d36ff7ba6..2e3c5a394e79 100644
@@ -308,12 +308,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg)
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -323,44 +317,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -368,7 +324,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	cpu_data(id).set = 1;
 }
 
@@ -1371,7 +1327,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
Quoted 8 lines

arch/x86/kernel/smpboot.c void __init native_smp_prepare_boot_cpu(void) · line 1342

Quoted 2 lines
arch/x86/kernel/smpboot.c +0 -0
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

I'll see if I can figure out a way to test that. Does 947134d9b00f ("x86/smpboot: Do not use smp_num_siblings in __max_logical_packages calculation") help?

P.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw

Re: [v6,3/3] x86/smpboot: Fix __max_logical_packages estimate

In reply to Simon Gaiser
Quoted 125 lines · Simon Gaiser

Prarit Bhargava:

Quoted 124 lines

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

ie) On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +10 -45
index 838d36ff7ba6..2e3c5a394e79 100644
@@ -308,12 +308,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg)
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -323,44 +317,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -368,7 +324,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	cpu_data(id).set = 1;
 }
 
@@ -1371,7 +1327,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
Quoted 8 lines

arch/x86/kernel/smpboot.c void __init native_smp_prepare_boot_cpu(void) · line 1342

Quoted 2 lines
arch/x86/kernel/smpboot.c +0 -0
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

I'll see if I can figure out a way to test that. Does 947134d9b00f ("x86/smpboot: Do not use smp_num_siblings in __max_logical_packages calculation") help?

P.

#12 lore raw

Re: [v6, 3/3] x86/smpboot: Fix __max_logical_packages estimate

Prarit Bhargava:

Quoted 135 lines · Simon Gaiser
Quoted 125 lines

Prarit Bhargava:

Quoted 124 lines

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

ie) On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +10 -45
index 838d36ff7ba6..2e3c5a394e79 100644
@@ -308,12 +308,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg)
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -323,44 +317,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -368,7 +324,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	cpu_data(id).set = 1;
 }
 
@@ -1371,7 +1327,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
Quoted 8 lines

arch/x86/kernel/smpboot.c void __init native_smp_prepare_boot_cpu(void) · line 1342

Quoted 2 lines
arch/x86/kernel/smpboot.c +0 -0
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

Quoted 1 line

I'll see if I can figure out a way to test that.

Ok, thanks. If you need some logs, or if I should test something just ask.

Quoted 3 lines

Does 947134d9b00f
("x86/smpboot: Do not use smp_num_siblings in __max_logical_packages calculation") help?

No.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [v6,3/3] x86/smpboot: Fix __max_logical_packages estimate

In reply to Prarit Bhargava

Prarit Bhargava:

Quoted 135 lines · Simon Gaiser
Quoted 125 lines

Prarit Bhargava:

Quoted 124 lines

A system booted with a small number of cores enabled per package panics because the estimate of __max_logical_packages is too low. This occurs when the total number of active cores across all packages is less than the maximum core count for a single package.

ie) On a 4 package system with 20 cores/package where only 4 cores are enabled on each package, the value of __max_logical_packages is calculated as DIV_ROUND_UP(16 / 20) = 1 and not 4.

Calculate __max_logical_packages after the cpu enumeration has completed. Use the boot cpu's data to extrapolate the number of packages.

Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Piotr Luc <[email protected]>
Cc Kan Liang <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Prarit Bhargava <[email protected]>
Cc Arvind Yadav <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc He Chen <[email protected]>
Cc Mathias Krause <[email protected]>
Cc Tim Chen <[email protected]>
Cc Vitaly Kuznetsov <[email protected]>
---
 arch/x86/kernel/smpboot.c | 55 +++++++++--------------------------------------
 1 file changed, 10 insertions(+), 45 deletions(-)
arch/x86/kernel/smpboot.c +10 -45
index 838d36ff7ba6..2e3c5a394e79 100644
@@ -308,12 +308,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	if (new >= 0)
 		goto found;
 
-	if (logical_packages >= __max_logical_packages) {
-		pr_warn("Package %u of CPU %u exceeds BIOS package data %u.\n",
-			logical_packages, cpu, __max_logical_packages);
-		return -ENOSPC;
-	}
-
 	new = logical_packages++;
 	if (new != pkg)
 		pr_info("CPU %u Converting physical %u to logical package %u\n",
@@ -323,44 +317,6 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
 	return 0;
 }
 
-static void __init smp_init_package_map(struct cpuinfo_x86 *c, unsigned int cpu)
-{
-	unsigned int ncpus;
-
-	/*
-	 * Today neither Intel nor AMD support heterogenous systems. That
-	 * might change in the future....
-	 *
-	 * While ideally we'd want '* smp_num_siblings' in the below @ncpus
-	 * computation, this won't actually work since some Intel BIOSes
-	 * report inconsistent HT data when they disable HT.
-	 *
-	 * In particular, they reduce the APIC-IDs to only include the cores,
-	 * but leave the CPUID topology to say there are (2) siblings.
-	 * This means we don't know how many threads there will be until
-	 * after the APIC enumeration.
-	 *
-	 * By not including this we'll sometimes over-estimate the number of
-	 * logical packages by the amount of !present siblings, but this is
-	 * still better than MAX_LOCAL_APIC.
-	 *
-	 * We use total_cpus not nr_cpu_ids because nr_cpu_ids can be limited
-	 * on the command line leading to a similar issue as the HT disable
-	 * problem because the hyperthreads are usually enumerated after the
-	 * primary cores.
-	 */
-	ncpus = boot_cpu_data.x86_max_cores;
-	if (!ncpus) {
-		pr_warn("x86_max_cores == zero !?!?");
-		ncpus = 1;
-	}
-
-	__max_logical_packages = DIV_ROUND_UP(total_cpus, ncpus);
-	pr_info("Max logical packages: %u\n", __max_logical_packages);
-
-	topology_update_package_map(c->phys_proc_id, cpu);
-}
-
 void __init smp_store_boot_cpu_info(void)
 {
 	int id = 0; /* CPU 0 */
@@ -368,7 +324,7 @@ void __init smp_store_boot_cpu_info(void)
 
 	*c = boot_cpu_data;
 	c->cpu_index = id;
-	smp_init_package_map(c, id);
+	topology_update_package_map(c->phys_proc_id, id);
 	cpu_data(id).set = 1;
 }
 
@@ -1371,7 +1327,16 @@ void __init native_smp_prepare_boot_cpu(void)
 
 void __init native_smp_cpus_done(unsigned int max_cpus)
 {
+	int ncpus;
+
 	pr_debug("Boot done\n");
+	/*
+	 * Today neither Intel nor AMD support heterogenous systems so
+	 * extrapolate the boot cpu's data to all packages.
+	 */
+	ncpus = cpu_data(0).booted_cores * smp_num_siblings;
+	__max_logical_packages = DIV_ROUND_UP(nr_cpu_ids, ncpus);
+	pr_info("Max logical packages: %u\n", __max_logical_packages);
 
Quoted 8 lines

arch/x86/kernel/smpboot.c void __init native_smp_prepare_boot_cpu(void) · line 1342

Quoted 2 lines
arch/x86/kernel/smpboot.c +0 -0
 	if (x86_has_numa_in_package)
 		set_sched_topology(x86_numa_in_package_topology);

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

Quoted 1 line

I'll see if I can figure out a way to test that.

Ok, thanks. If you need some logs, or if I should test something just ask.

Quoted 3 lines

Does 947134d9b00f
("x86/smpboot: Do not use smp_num_siblings in __max_logical_packages calculation") help?

No.

#14 lore raw

Re: [v6, 3/3] x86/smpboot: Fix __max_logical_packages estimate

Quoted 2 lines · Simon Gaiser

Prarit Bhargava:

Quoted 1 line · Simon Gaiser

<snip>

Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

I'll see if I can figure out a way to test that.

Ok, thanks. If you need some logs, or if I should test something just ask.

Thanks, I may send you a patch off-list to test.

Quoted 6 lines
Quoted 3 lines

Does 947134d9b00f
("x86/smpboot: Do not use smp_num_siblings in __max_logical_packages calculation") help?

No.

Yeah, your description was absolutely correct. native_smp_cpus_done() is only called for HVM guests and not PV. I'm going to think about that and send you a patch in a bit.

P.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

Re: [v6,3/3] x86/smpboot: Fix __max_logical_packages estimate

In reply to Simon Gaiser
Quoted 2 lines · Simon Gaiser

Prarit Bhargava:

Quoted 1 line · Simon Gaiser

<snip>

Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

This breaks booting as Xen PV domain for me. The problem seems to be that native_smp_cpus_done() is never called on a PV domain. So __max_logical_packages is uninitialized and this leads to a NULL pointer dereference in coretemp.

I'll see if I can figure out a way to test that.

Ok, thanks. If you need some logs, or if I should test something just ask.

Thanks, I may send you a patch off-list to test.

Quoted 6 lines
Quoted 3 lines

Does 947134d9b00f
("x86/smpboot: Do not use smp_num_siblings in __max_logical_packages calculation") help?

No.

Yeah, your description was absolutely correct. native_smp_cpus_done() is only called for HVM guests and not PV. I'm going to think about that and send you a patch in a bit.

P.