Skip to contentSkip to conversation

mainline boot regression on AMD Stoney Ridge Chromebooks

REGRESSION Applied Laura Nao 4 participants 37 messages Mar 22, 2024 – Apr 12, 2024

Subsystems (2)
lore
Share & tools
lore

Laura Nao

author
#0 lore raw

Hello,

KernelCI has identified a mainline boot regression [1] on the following AMD Stoney Ridge Chromebooks (grunt family), between v6.8 (e8f897) and v6.8-1185-g855684c7d938 (855684):
- Acer Chromebook Spin 311 R721T (codename kasumi360)
- HP Chromebook 14 (codename careena)
- HP Chromebook 11A G6 EE (codename barla)

The kernel doesn't boot at all and nothing is reported on the serial console after "Starting kernel ...". The issue is still present on the latest mainline revision.
The defconfig used by KernelCI for the boot tests can be found in [2].

Sending this report in order to track the regression while a fix is identified.

Thanks,

Laura

[1] https://linux.kernelci.org/test/case/id/65fca98e3883a392524c4380/
[2]
https://storage.kernelci.org/mainline/master/v6.8-11837-g2ac2b1665d3fb/x86_64/x86_64_defconfig%2Bx86-board/gcc-10/config/kernel.config

#regzbot introduced: v6.8..v6.8-1185-g855684c7d938

author
#1 lore raw

Hi Thomas,

I ran a manual bisection to track down the root cause for this regression and landed on the c749ce commit from this series:
https://lore.kernel.org/all/[email protected]/

Do you have any insight on this issue or any suggestion on how to effectively debug this?

Thank you!

Laura

#regzbot introduced: c749ce393b

author
#2 lore raw
Quoted 6 lines

Hi Thomas,

I ran a manual bisection to track down the root cause for this regression and landed on the c749ce commit from this series:
https://lore.kernel.org/all/[email protected]/

Quoted 6 lines

Do you have any insight on this issue or any suggestion on how to effectively debug this?

Thank you!

Laura

Hi again,

I actually forgot to quote the previous message describing the issue:

Quoted 15 lines

KernelCI has identified a mainline boot regression [1] on the following AMD Stoney Ridge Chromebooks (grunt family), between v6.8 (e8f897) and v6.8-1185-g855684c7d938 (855684):
- Acer Chromebook Spin 311 R721T (codename kasumi360)
- HP Chromebook 14 (codename careena)
- HP Chromebook 11A G6 EE (codename barla)

The kernel doesn't boot at all and nothing is reported on the serial console after "Starting kernel ...". The issue is still present on the latest mainline revision.
The defconfig used by KernelCI for the boot tests can be found in [2].

Sending this report in order to track the regression while a fix is identified.

Quoted 7 lines

Thanks,

Laura

[1] https://linux.kernelci.org/test/case/id/65fca98e3883a392524c4380/
[2]
https://storage.kernelci.org/mainline/master/v6.8-11837-g2ac2b1665d3fb/x86_64/x86_64_defconfig%2Bx86-board/gcc-10/config/kernel.config

sorry about that!

Best,

Laura

#3 lore raw

[/me added x86 team]

Quoted 5 lines · Laura Nao
Quoted 5 lines

I ran a manual bisection to track down the root cause for this regression and landed on the c749ce
commit from this series:
https://lore.kernel.org/all/[email protected]/

FWIW, that commit is c749ce393b8fe9 ("x86/cpu: Use common topology code for AMD") from tglx that was part of the "x86/cpu: Rework topology evaluation" series.

Quoted 4 lines
Quoted 4 lines

Do you have any insight on this issue or any suggestion on how to effectively debug this?

Thank you!

Hmmm, it looks like this did not make any progress. Thomas, did this fall through the cracks due to Easter, or is this this on your todo list?

Or was there some progress and I just missed it?

Laura Nao, I assume the problem is still happening?

FWIW, this was the initial problem description:

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

KernelCI has identified a mainline boot regression [1] on the following AMD Stoney Ridge Chromebooks (grunt family), between v6.8 (e8f897) and v6.8-1185-g855684c7d938 (855684):
- Acer Chromebook Spin 311 R721T (codename kasumi360)
- HP Chromebook 14 (codename careena)
- HP Chromebook 11A G6 EE (codename barla)

The kernel doesn't boot at all and nothing is reported on the serial console after "Starting kernel ...". The issue is still present on the latest mainline revision.
The defconfig used by KernelCI for the boot tests can be found in [2].

Sending this report in order to track the regression while a fix is identified.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Thanks,

Laura

[1] https://linux.kernelci.org/test/case/id/65fca98e3883a392524c4380/
[2]
https://storage.kernelci.org/mainline/master/v6.8-11837-g2ac2b1665d3fb/x86_64/x86_64_defconfig%2Bx86-board/gcc-10/config/kernel.config

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

author
#4 lore raw

Hi Thorsten,

Quoted 16 lines · Linux regression tracking (Thorsten Leemhuis)

[/me added x86 team]

Quoted 5 lines · Laura Nao
Quoted 5 lines

I ran a manual bisection to track down the root cause for this regression and landed on the c749ce
commit from this series:
https://lore.kernel.org/all/[email protected]/

FWIW, that commit is c749ce393b8fe9 ("x86/cpu: Use common topology code for AMD") from tglx that was part of the "x86/cpu: Rework topology evaluation" series.

Quoted 3 lines
Quoted 3 lines

Do you have any insight on this issue or any suggestion on how to effectively debug this?

Quoted 8 lines
Quoted 1 line
Quoted 1 line

Thank you!

Hmmm, it looks like this did not make any progress. Thomas, did this fall through the cracks due to Easter, or is this this on your todo list?

Or was there some progress and I just missed it?

Laura Nao, I assume the problem is still happening?

Yes, I confirm the issue is still happening.

See e.g. KernelCI report for next-20240403:

https://linux.kernelci.org/test/plan/id/660dbeed148e1a8a284c4372/

Quoted 34 lines

FWIW, this was the initial problem description:

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

KernelCI has identified a mainline boot regression [1] on the following AMD Stoney Ridge Chromebooks (grunt family), between v6.8 (e8f897) and v6.8-1185-g855684c7d938 (855684):
- Acer Chromebook Spin 311 R721T (codename kasumi360)
- HP Chromebook 14 (codename careena)
- HP Chromebook 11A G6 EE (codename barla)

The kernel doesn't boot at all and nothing is reported on the serial console after "Starting kernel ...". The issue is still present on the latest mainline revision.
The defconfig used by KernelCI for the boot tests can be found in [2].

Sending this report in order to track the regression while a fix is identified.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Thanks,

Laura

[1] https://linux.kernelci.org/test/case/id/65fca98e3883a392524c4380/
[2]
https://storage.kernelci.org/mainline/master/v6.8-11837-g2ac2b1665d3fb/x86_64/x86_64_defconfig%2Bx86-board/gcc-10/config/kernel.config

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

#5 lore raw
In reply to Linux regression tracking (Thorsten Leemhuis)
Quoted 14 lines · Linux regression tracking
Quoted 5 lines · Laura Nao
Quoted 5 lines

I ran a manual bisection to track down the root cause for this regression and landed on the c749ce
commit from this series:
https://lore.kernel.org/all/[email protected]/

FWIW, that commit is c749ce393b8fe9 ("x86/cpu: Use common topology code for AMD") from tglx that was part of the "x86/cpu: Rework topology evaluation" series.

Quoted 3 lines
Quoted 3 lines

Do you have any insight on this issue or any suggestion on how to effectively debug this?

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Thank you!

Hmmm, it looks like this did not make any progress. Thomas, did this fall through the cracks due to Easter, or is this this on your todo list?

Or was there some progress and I just missed it?

No. It's in my easter eggs basket, but I need to eat all the chocolate eggs first to get to this.

I'll have a look later today.

Thanks,

tglx

#6 lore raw

Laura!

Quoted 11 lines · Thomas Gleixner
Quoted 10 lines · Linux regression tracking
Quoted 5 lines · Laura Nao
Quoted 5 lines

I ran a manual bisection to track down the root cause for this regression and landed on the c749ce
commit from this series:
https://lore.kernel.org/all/[email protected]/

FWIW, that commit is c749ce393b8fe9 ("x86/cpu: Use common topology code for AMD") from tglx that was part of the "x86/cpu: Rework topology evaluation" series.

Can you please provide a boot dmesg from a 6.8 kernel and the output of 'cpuid' ?

Thanks,

tglx

author
#7 lore raw

Hi Thomas,

Quoted 11 lines · Thomas Gleixner

Laura!

Quoted 8 lines · Thomas Gleixner
Quoted 7 lines · Linux regression tracking
Quoted 5 lines · Laura Nao
Quoted 5 lines

I ran a manual bisection to track down the root cause for this regression and landed on the c749ce
commit from this series:
https://lore.kernel.org/all/[email protected]/

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

FWIW, that commit is c749ce393b8fe9 ("x86/cpu: Use common topology code for AMD") from tglx that was part of the "x86/cpu: Rework topology evaluation" series.

Can you please provide a boot dmesg from a 6.8 kernel and the output of 'cpuid' ?

Sure, here's the output from an Acer Chromebook Spin 311 R721T device:
- kernel log: https://pastebin.com/raw/NNbeJupE
- cpuid output: https://pastebin.com/raw/ThmGqnqG

This is with kernel v6.8 (e8f897f4afef0031fe618a8e94127a0934896aba), let me know if you need the output from a different revision.

Thank you!

Laura

#8 lore raw

Laura!

Quoted 7 lines · Laura Nao
Quoted 2 lines · Thomas Gleixner

Can you please provide a boot dmesg from a 6.8 kernel and the output of 'cpuid' ?

Sure, here's the output from an Acer Chromebook Spin 311 R721T device:
- kernel log: https://pastebin.com/raw/NNbeJupE
- cpuid output: https://pastebin.com/raw/ThmGqnqG

Thanks for the quick response.

Can you please provide 'cpuid -r' output too as 'cpuid' does a few tweaks to the raw data and it's hard to match it back to the code.

Thanks,

tglx

#9 lore raw

Laura!

Quoted 2 lines · Thomas Gleixner

Can you please provide 'cpuid -r' output too as 'cpuid' does a few tweaks to the raw data and it's hard to match it back to the code.

Don't bother. I think I figured it out.

Can you please test the patch below?

Thanks,

tglx
---

arch/x86/kernel/cpu/topology_amd.c +17 -5
index 1a8b3ad493af..0d91a04b1741 100644
@@ -29,7 +29,17 @@ static bool parse_8000_0008(struct topo_scan *tscan)
 	if (!sft)
 		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
 	return true;
 }
 
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_0xb)
 	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
 	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
 	 */
 	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
 		unsigned int nthreads = leaf.core_nthreads + 1;
 
 		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
#10 lore raw
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines · Thomas Gleixner

Can you please provide 'cpuid -r' output too as 'cpuid' does a few tweaks to the raw data and it's hard to match it back to the code.

Don't bother. I think I figured it out.

Can you please test the patch below?

Bah. Won't help. Let me dig deeper.

#11 lore raw
Quoted 10 lines · Thomas Gleixner
Quoted 7 lines · Thomas Gleixner
Quoted 2 lines · Thomas Gleixner

Can you please provide 'cpuid -r' output too as 'cpuid' does a few tweaks to the raw data and it's hard to match it back to the code.

Don't bother. I think I figured it out.

Can you please test the patch below?

Bah. Won't help. Let me dig deeper.

Can I get the /proc/cpuinfo output please?

author
#12 lore raw

Hi Thomas,

Quoted 7 lines · Thomas Gleixner
Quoted 5 lines · Thomas Gleixner
Quoted 4 lines · Thomas Gleixner
Quoted 2 lines · Thomas Gleixner

Can you please provide 'cpuid -r' output too as 'cpuid' does a few tweaks to the raw data and it's hard to match it back to the code.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Don't bother. I think I figured it out.

Can you please test the patch below?

Bah. Won't help. Let me dig deeper.

Can I get the /proc/cpuinfo output please?

Here are more logs from the same device running v6.8:
- cpuid -r: https://pastebin.com/raw/YmDrnw06
- cpuinfo: https://pastebin.com/raw/Ki0vwdDE

Thanks for looking into this!

Laura

#13 lore raw

Laura!

Quoted 6 lines · Laura Nao
Quoted 1 line · Thomas Gleixner

Can I get the /proc/cpuinfo output please?

Here are more logs from the same device running v6.8:
- cpuid -r: https://pastebin.com/raw/YmDrnw06
- cpuinfo: https://pastebin.com/raw/Ki0vwdDE

Thanks for the info. I'm still not seing why that wouldn't work.

As the machine seems to have a serial port, can you please replace 'earlycon' with 'earlyprintk=ttyS0,115200' on the command line and see whether that gives some output. I'm not sure that the current 'earlycon' has the same effect.

Thanks,

Thomas

author
#14 lore raw
Quoted 10 lines · Thomas Gleixner

Laura!

Quoted 6 lines · Laura Nao
Quoted 1 line · Thomas Gleixner

Can I get the /proc/cpuinfo output please?

Here are more logs from the same device running v6.8:
- cpuid -r: https://pastebin.com/raw/YmDrnw06
- cpuinfo: https://pastebin.com/raw/Ki0vwdDE

Quoted 7 lines

Thanks for the info. I'm still not seing why that wouldn't work.

As the machine seems to have a serial port, can you please replace 'earlycon' with 'earlyprintk=ttyS0,115200' on the command line and see whether that gives some output. I'm not sure that the current 'earlycon' has the same effect.

Thanks for the hint - replacing earlycon with earlyprintk=ttyS0,115200 doesn't seem to help unfortunately, I still get no output at all.

Not sure if it's really relevant to mention, but when booting a v6.8 kernel the first stage bootloader (i.e. coreboot in this case) spits out a few lines before the kernel takes over:

coreboot-v1.9308_26_0.0.22-6034-gd586c7b122 Wed Jul 15 17:51:44 UTC 2020 smm starting... SMI# #0
Chrome EC: Set SMI mask to 0x0000000000000000
Chrome EC: UHEPI supported
Clearing pending EC events. Error code 1 is expected.
EC returned error result code 9
Chrome EC: Set SCI mask to 0x00000000142609fb

When booting e.g. next-20240404 not even these lines are reported after 'Starting Kernel...'. Not sure if that suggests anything significant though.

For the sake of completeness, I'm leaving here links to the full test jobs run in our Collabora LAVA lab, to include everything that's reported on the serial console after the device is powered on:
- v6.8 test job: https://lava.collabora.dev/scheduler/job/13265735#L279
- next-20240404 test job w/ 'earlyprintk=ttyS0,115200' in cmdline:
https://lava.collabora.dev/scheduler/job/13266992#L279

I also tried applying the patch you attached above on top of next-20240404 but it doesn't seem to make any difference unfortunately.

Best,

Laura

#15 lore raw
Quoted 2 lines · Laura Nao

I also tried applying the patch you attached above on top of next-20240404 but it doesn't seem to make any difference unfortunately.

As I said it won't help, but it fixes some other issue.

I'm still puzzled and trying to make sense of this.

Can you please boot a working kernel and provide the output of:

# rdmsr 0xc0011005

Thanks,

tglx

author
#16 lore raw
Quoted 5 lines · Thomas Gleixner

Can you please boot a working kernel and provide the output of:

# rdmsr 0xc0011005

Sure:

# rdmsr 0xc0011005
2fabbfff2fd3fbff

Thanks,

Laura

#17 lore raw
Quoted 8 lines · Laura Nao
Quoted 4 lines · Thomas Gleixner

Can you please boot a working kernel and provide the output of:

# rdmsr 0xc0011005
# rdmsr 0xc0011005
2fabbfff2fd3fbff

Ok. So the CPU does not advertise FEATURE_TOPOEXT which is consistent with CPUID.

Now I'm even more puzzled because then the patch I gave you should make a difference. I need to run some errands now, I will provide a debug patch later this evening.

Thanks,

tglx

#18 lore raw

Laura!

Quoted 10 lines · Thomas Gleixner
Quoted 8 lines · Laura Nao
Quoted 4 lines · Thomas Gleixner

Can you please boot a working kernel and provide the output of:

# rdmsr 0xc0011005
# rdmsr 0xc0011005
2fabbfff2fd3fbff
Quoted 6 lines

Ok. So the CPU does not advertise FEATURE_TOPOEXT which is consistent with CPUID.

Now I'm even more puzzled because then the patch I gave you should make a difference. I need to run some errands now, I will provide a debug patch later this evening.

Hmm. No real good idea yet.

Can you please checkout

    c749ce393b8f ("x86/cpu: Use common topology code for AMD")

and apply the patch further up in the thread and see whether that boots. We'll take it from there.

Thanks,

tglx

author
#19 lore raw

Hi Thomas,

Quoted 20 lines · Thomas Gleixner

Laura!

Quoted 10 lines · Thomas Gleixner
Quoted 8 lines · Laura Nao
Quoted 4 lines · Thomas Gleixner

Can you please boot a working kernel and provide the output of:

# rdmsr 0xc0011005
# rdmsr 0xc0011005
2fabbfff2fd3fbff
Quoted 6 lines

Ok. So the CPU does not advertise FEATURE_TOPOEXT which is consistent with CPUID.

Now I'm even more puzzled because then the patch I gave you should make a difference. I need to run some errands now, I will provide a debug patch later this evening.

Quoted 9 lines

Hmm. No real good idea yet.

Can you please checkout

     c749ce393b8f ("x86/cpu: Use common topology code for AMD")

and apply the patch further up in the thread and see whether that boots. We'll take it from there.

Just tried that and it doesn't boot, nothing on the serial console (I kept earlyprintk=ttyS0,115200 in the cmdline).

Test job for reference: https://lava.collabora.dev/scheduler/job/13297561

Thanks,

Laura

#20 lore raw

Laura!

Quoted 2 lines · Laura Nao

Just tried that and it doesn't boot, nothing on the serial console (I kept earlyprintk=ttyS0,115200 in the cmdline).

Thanks for trying. Now let's take a small step back.

Please reset the tree to:

      ace278e7eca6 ("x86/smpboot: Teach it about topo.amd_node_id")

That's the commit right before switching over and according to your bisect it works. Now apply the patch below, which just runs the new topology scan function but discards the result.

Thanks,

tglx
---

arch/x86/kernel/cpu/topology_amd.c +17 -5
@@ -29,7 +29,17 @@ static bool parse_8000_0008(struct topo_
 	if (!sft)
 		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
 	return true;
 }
 
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_
 	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
 	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
 	 */
 	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
 		unsigned int nthreads = leaf.core_nthreads + 1;
 
 		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
arch/x86/kernel/cpu/topology_common.c +4 -0
@@ -133,6 +133,10 @@ static void parse_topology(struct topo_s
 	tscan->ebx1_nproc_shift = get_count_order(ebx.nproc);
 
 	switch (c->x86_vendor) {
+	case X86_VENDOR_AMD:
+		if (IS_ENABLED(CONFIG_CPU_SUP_AMD))
+			cpu_parse_topology_amd(tscan);
+		break;
 	case X86_VENDOR_CENTAUR:
 	case X86_VENDOR_ZHAOXIN:
 		parse_legacy(tscan);
author
#21 lore raw

Hi Thomas,

Quoted 8 lines · Thomas Gleixner

Laura!

Quoted 2 lines · Laura Nao

Just tried that and it doesn't boot, nothing on the serial console (I kept earlyprintk=ttyS0,115200 in the cmdline).

Thanks for trying. Now let's take a small step back.

Quoted 8 lines

Please reset the tree to:

       ace278e7eca6 ("x86/smpboot: Teach it about topo.amd_node_id")

That's the commit right before switching over and according to your bisect it works. Now apply the patch below, which just runs the new topology scan function but discards the result.

The ace278e7eca6 revision with the patch you provided boots correctly.

Reference test job: https://lava.collabora.dev/scheduler/job/13311035#L1894

Thanks,

Laura

#22 lore raw

Laura!

Quoted 6 lines · Laura Nao
Quoted 3 lines · Thomas Gleixner

That's the commit right before switching over and according to your bisect it works. Now apply the patch below, which just runs the new topology scan function but discards the result.

The ace278e7eca6 revision with the patch you provided boots correctly.

Progress :)

Can you please replace that patch with the one below?

Thanks,

tglx
---

arch/x86/kernel/cpu/common.c +7 -0
@@ -1616,6 +1616,13 @@ void __init early_cpu_init(void)
 #endif
 	}
 	early_identify_cpu(&boot_cpu_data);
+
+	pr_info("Max cores: %u\n", boot_cpu_data.x86_max_cores);
+	pr_info("pkg %u die %u core %u nid %u\n", boot_cpu_data.topo.pkg_id,
+		boot_cpu_data.topo.die_id, boot_cpu_data.topo.core_id,
+		boot_cpu_data.topo.amd_node_id);
+	pr_info("SNS %u\n", smp_num_siblings);
+	pr_info("NPP %u MDPP %u\n", __amd_nodes_per_pkg, __max_die_per_package);
 }
 
 static bool detect_null_seg_behavior(void)
arch/x86/kernel/cpu/topology_amd.c +17 -5
@@ -29,7 +29,17 @@ static bool parse_8000_0008(struct topo_
 	if (!sft)
 		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
 	return true;
 }
 
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_
 	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
 	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
 	 */
 	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
 		unsigned int nthreads = leaf.core_nthreads + 1;
 
 		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
arch/x86/kernel/cpu/topology_common.c +15 -3
@@ -133,6 +133,10 @@ static void parse_topology(struct topo_s
 	tscan->ebx1_nproc_shift = get_count_order(ebx.nproc);
 
 	switch (c->x86_vendor) {
+	case X86_VENDOR_AMD:
+		if (IS_ENABLED(CONFIG_CPU_SUP_AMD))
+			cpu_parse_topology_amd(tscan);
+		break;
 	case X86_VENDOR_CENTAUR:
 	case X86_VENDOR_ZHAOXIN:
 		parse_legacy(tscan);
@@ -162,6 +166,9 @@ static void topo_set_ids(struct topo_sca
 
 	if (c->x86_vendor == X86_VENDOR_AMD)
 		cpu_topology_fixup_amd(tscan);
+
+	pr_info("pkg %u die %u core %u nid %u\n", c->topo.pkg_id, c->topo.die_id,
+		c->topo.core_id, c->topo.amd_node_id);
 }
 
 static void topo_set_max_cores(struct topo_scan *tscan)
@@ -175,6 +182,7 @@ static void topo_set_max_cores(struct to
 	 */
 	tscan->c->x86_max_cores = tscan->dom_ncpus[TOPO_DIEGRP_DOMAIN] >>
 		x86_topo_system.dom_shifts[TOPO_SMT_DOMAIN];
+	pr_info("Max cores: %u\n", tscan->c->x86_max_cores);
 }
 
 void cpu_parse_topology(struct cpuinfo_x86 *c)
@@ -215,18 +223,19 @@ void __init cpu_init_topology(struct cpu
 
 	parse_topology(&tscan, true);
 
-	if (!topo_is_converted(c))
-		return;
-
 	/* Copy the shift values and calculate the unit sizes. */
 	memcpy(x86_topo_system.dom_shifts, tscan.dom_shifts, sizeof(x86_topo_system.dom_shifts));
 
 	dom = TOPO_SMT_DOMAIN;
 	x86_topo_system.dom_size[dom] = 1U << x86_topo_system.dom_shifts[dom];
+	pr_info("Dom %u Sft: %u Sz: %u\n", dom, x86_topo_system.dom_size[dom],
+		x86_topo_system.dom_shifts[dom]);
 
 	for (dom++; dom < TOPO_MAX_DOMAIN; dom++) {
 		sft = x86_topo_system.dom_shifts[dom] - x86_topo_system.dom_shifts[dom - 1];
 		x86_topo_system.dom_size[dom] = 1U << sft;
+		pr_info("Dom %u Sft: %u Sz: %u\n", dom, x86_topo_system.dom_size[dom],
+			x86_topo_system.dom_shifts[dom]);
 	}
 
 	topo_set_ids(&tscan);
@@ -238,6 +247,7 @@ void __init cpu_init_topology(struct cpu
 	 * changes further down the road to get it right during early boot.
 	 */
 	smp_num_siblings = tscan.dom_ncpus[TOPO_SMT_DOMAIN];
+	pr_info("SNS %u\n", smp_num_siblings);
 
 	/*
 	 * Neither it's clear whether there are as many dies as the APIC
@@ -252,4 +262,6 @@ void __init cpu_init_topology(struct cpu
 	 */
 	if (c->x86_vendor == X86_VENDOR_AMD || c->x86_vendor == X86_VENDOR_HYGON)
 		__amd_nodes_per_pkg = __max_die_per_package = tscan.amd_nodes_per_pkg;
+
+	pr_info("NPP %u MDPP %u\n", __amd_nodes_per_pkg, __max_die_per_package);
 }
author
#23 lore raw

Hi Thomas,

Quoted 12 lines · Thomas Gleixner

Laura!

Quoted 6 lines · Laura Nao
Quoted 3 lines · Thomas Gleixner

That's the commit right before switching over and according to your bisect it works. Now apply the patch below, which just runs the new topology scan function but discards the result.

The ace278e7eca6 revision with the patch you provided boots correctly.

Progress :)

Indeed :)

Quoted 1 line

Can you please replace that patch with the one below?

So, with this patch applied on top of ace278e7eca6 the kernel doesn't boot anymore - reference test job:
https://lava.collabora.dev/scheduler/job/13324010

I see the only change between the second and third patch you provided, besides the debug prints, is:

- if (!topo_is_converted(c))
- return;
-

Printing the debug information without this probably doesn't really help, but just in case it's useful: I tried excluding the change above from the patch while leaving everything else unchanged - reference test job:
https://lava.collabora.dev/scheduler/job/13324298 (also pasted the kernel log here for easier consultation: https://pastebin.com/raw/TQBDvCah)

Hope this helps,

Laura

Quoted 130 lines

Thanks,

tglx
---

arch/x86/kernel/cpu/common.c +7 -0
@@ -1616,6 +1616,13 @@ void __init early_cpu_init(void)
  #endif
  	}
  	early_identify_cpu(&boot_cpu_data);
+
+	pr_info("Max cores: %u\n", boot_cpu_data.x86_max_cores);
+	pr_info("pkg %u die %u core %u nid %u\n", boot_cpu_data.topo.pkg_id,
+		boot_cpu_data.topo.die_id, boot_cpu_data.topo.core_id,
+		boot_cpu_data.topo.amd_node_id);
+	pr_info("SNS %u\n", smp_num_siblings);
+	pr_info("NPP %u MDPP %u\n", __amd_nodes_per_pkg, __max_die_per_package);
  }
 
  static bool detect_null_seg_behavior(void)
arch/x86/kernel/cpu/topology_amd.c +17 -5
@@ -29,7 +29,17 @@ static bool parse_8000_0008(struct topo_
  	if (!sft)
  		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
  	return true;
  }
 
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_
  	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
  	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
  	 */
  	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
  		unsigned int nthreads = leaf.core_nthreads + 1;
 
  		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
arch/x86/kernel/cpu/topology_common.c +15 -3
@@ -133,6 +133,10 @@ static void parse_topology(struct topo_s
  	tscan->ebx1_nproc_shift = get_count_order(ebx.nproc);
 
  	switch (c->x86_vendor) {
+	case X86_VENDOR_AMD:
+		if (IS_ENABLED(CONFIG_CPU_SUP_AMD))
+			cpu_parse_topology_amd(tscan);
+		break;
  	case X86_VENDOR_CENTAUR:
  	case X86_VENDOR_ZHAOXIN:
  		parse_legacy(tscan);
@@ -162,6 +166,9 @@ static void topo_set_ids(struct topo_sca
 
  	if (c->x86_vendor == X86_VENDOR_AMD)
  		cpu_topology_fixup_amd(tscan);
+
+	pr_info("pkg %u die %u core %u nid %u\n", c->topo.pkg_id, c->topo.die_id,
+		c->topo.core_id, c->topo.amd_node_id);
  }
 
  static void topo_set_max_cores(struct topo_scan *tscan)
@@ -175,6 +182,7 @@ static void topo_set_max_cores(struct to
  	 */
  	tscan->c->x86_max_cores = tscan->dom_ncpus[TOPO_DIEGRP_DOMAIN] >>
  		x86_topo_system.dom_shifts[TOPO_SMT_DOMAIN];
+	pr_info("Max cores: %u\n", tscan->c->x86_max_cores);
  }
 
  void cpu_parse_topology(struct cpuinfo_x86 *c)
@@ -215,18 +223,19 @@ void __init cpu_init_topology(struct cpu
 
  	parse_topology(&tscan, true);
 
-	if (!topo_is_converted(c))
-		return;
-
  	/* Copy the shift values and calculate the unit sizes. */
  	memcpy(x86_topo_system.dom_shifts, tscan.dom_shifts, sizeof(x86_topo_system.dom_shifts));
 
  	dom = TOPO_SMT_DOMAIN;
  	x86_topo_system.dom_size[dom] = 1U << x86_topo_system.dom_shifts[dom];
+	pr_info("Dom %u Sft: %u Sz: %u\n", dom, x86_topo_system.dom_size[dom],
+		x86_topo_system.dom_shifts[dom]);
 
  	for (dom++; dom < TOPO_MAX_DOMAIN; dom++) {
  		sft = x86_topo_system.dom_shifts[dom] - x86_topo_system.dom_shifts[dom - 1];
  		x86_topo_system.dom_size[dom] = 1U << sft;
+		pr_info("Dom %u Sft: %u Sz: %u\n", dom, x86_topo_system.dom_size[dom],
+			x86_topo_system.dom_shifts[dom]);
  	}
 
  	topo_set_ids(&tscan);
@@ -238,6 +247,7 @@ void __init cpu_init_topology(struct cpu
  	 * changes further down the road to get it right during early boot.
  	 */
  	smp_num_siblings = tscan.dom_ncpus[TOPO_SMT_DOMAIN];
+	pr_info("SNS %u\n", smp_num_siblings);
 
  	/*
  	 * Neither it's clear whether there are as many dies as the APIC
@@ -252,4 +262,6 @@ void __init cpu_init_topology(struct cpu
  	 */
  	if (c->x86_vendor == X86_VENDOR_AMD || c->x86_vendor == X86_VENDOR_HYGON)
  		__amd_nodes_per_pkg = __max_die_per_package = tscan.amd_nodes_per_pkg;
+
+	pr_info("NPP %u MDPP %u\n", __amd_nodes_per_pkg, __max_die_per_package);
  }
#24 lore raw

Laura!

Review

Quoted 13 lines · Laura Nao
Quoted 1 line · Thomas Gleixner

Can you please replace that patch with the one below?

So, with this patch applied on top of ace278e7eca6 the kernel doesn't boot anymore - reference test job:
https://lava.collabora.dev/scheduler/job/13324010

I see the only change between the second and third patch you provided, besides the debug prints, is:

(quoted diff) +0 -3
-	if (!topo_is_converted(c))
-		return;
-

Right. So this limits the area to search significantly.

Quoted 8 lines

Printing the debug information without this probably doesn't really help, but just in case it's useful: I tried excluding the change above from the patch while leaving everything else unchanged - reference test job:
https://lava.collabora.dev/scheduler/job/13324298 (also pasted the kernel log here for easier consultation:
https://pastebin.com/raw/TQBDvCah)

Hope this helps,

It does. Good idea!

I just moved the exit check a bit so we should see the scan info. That should tell me what goes south.

Thanks,

tglx
---

arch/x86/kernel/cpu/common.c +7 -0
@@ -1616,6 +1616,13 @@ void __init early_cpu_init(void)
 #endif
 	}
 	early_identify_cpu(&boot_cpu_data);
+
+	pr_info("Max cores: %u\n", boot_cpu_data.x86_max_cores);
+	pr_info("pkg %u die %u core %u nid %u\n", boot_cpu_data.topo.pkg_id,
+		boot_cpu_data.topo.die_id, boot_cpu_data.topo.core_id,
+		boot_cpu_data.topo.amd_node_id);
+	pr_info("SNS %u\n", smp_num_siblings);
+	pr_info("NPP %u MDPP %u\n", __amd_nodes_per_pkg, __max_die_per_package);
 }
 
 static bool detect_null_seg_behavior(void)
arch/x86/kernel/cpu/topology_amd.c +17 -5
@@ -29,7 +29,17 @@ static bool parse_8000_0008(struct topo_
 	if (!sft)
 		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
 	return true;
 }
 
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_
 	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
 	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
 	 */
 	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
 		unsigned int nthreads = leaf.core_nthreads + 1;
 
 		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
arch/x86/kernel/cpu/topology_common.c +20 -3
@@ -133,6 +133,10 @@ static void parse_topology(struct topo_s
 	tscan->ebx1_nproc_shift = get_count_order(ebx.nproc);
 
 	switch (c->x86_vendor) {
+	case X86_VENDOR_AMD:
+		if (IS_ENABLED(CONFIG_CPU_SUP_AMD))
+			cpu_parse_topology_amd(tscan);
+		break;
 	case X86_VENDOR_CENTAUR:
 	case X86_VENDOR_ZHAOXIN:
 		parse_legacy(tscan);
@@ -162,6 +166,9 @@ static void topo_set_ids(struct topo_sca
 
 	if (c->x86_vendor == X86_VENDOR_AMD)
 		cpu_topology_fixup_amd(tscan);
+
+	pr_info("pkg %u die %u core %u nid %u\n", c->topo.pkg_id, c->topo.die_id,
+		c->topo.core_id, c->topo.amd_node_id);
 }
 
 static void topo_set_max_cores(struct topo_scan *tscan)
@@ -175,6 +182,7 @@ static void topo_set_max_cores(struct to
 	 */
 	tscan->c->x86_max_cores = tscan->dom_ncpus[TOPO_DIEGRP_DOMAIN] >>
 		x86_topo_system.dom_shifts[TOPO_SMT_DOMAIN];
+	pr_info("Max cores: %u\n", tscan->c->x86_max_cores);
 }
 
 void cpu_parse_topology(struct cpuinfo_x86 *c)
@@ -215,20 +223,26 @@ void __init cpu_init_topology(struct cpu
 
 	parse_topology(&tscan, true);
 
-	if (!topo_is_converted(c))
-		return;
-
 	/* Copy the shift values and calculate the unit sizes. */
 	memcpy(x86_topo_system.dom_shifts, tscan.dom_shifts, sizeof(x86_topo_system.dom_shifts));
 
 	dom = TOPO_SMT_DOMAIN;
 	x86_topo_system.dom_size[dom] = 1U << x86_topo_system.dom_shifts[dom];
+	pr_info("Dom %u Sft: %u Sz: %u\n", dom, x86_topo_system.dom_size[dom],
+		x86_topo_system.dom_shifts[dom]);
 
 	for (dom++; dom < TOPO_MAX_DOMAIN; dom++) {
 		sft = x86_topo_system.dom_shifts[dom] - x86_topo_system.dom_shifts[dom - 1];
 		x86_topo_system.dom_size[dom] = 1U << sft;
+		pr_info("Dom %u Sft: %u Sz: %u\n", dom, x86_topo_system.dom_size[dom],
+			x86_topo_system.dom_shifts[dom]);
 	}
 
+	pr_info("NPP %u\n", tscan.amd_nodes_per_pkg);
+
+	if (!topo_is_converted(c))
+		return;
+
 	topo_set_ids(&tscan);
 	topo_set_max_cores(&tscan);
 
@@ -238,6 +252,7 @@ void __init cpu_init_topology(struct cpu
 	 * changes further down the road to get it right during early boot.
 	 */
 	smp_num_siblings = tscan.dom_ncpus[TOPO_SMT_DOMAIN];
+	pr_info("SNS %u\n", smp_num_siblings);
 
 	/*
 	 * Neither it's clear whether there are as many dies as the APIC
@@ -252,4 +267,6 @@ void __init cpu_init_topology(struct cpu
 	 */
 	if (c->x86_vendor == X86_VENDOR_AMD || c->x86_vendor == X86_VENDOR_HYGON)
 		__amd_nodes_per_pkg = __max_die_per_package = tscan.amd_nodes_per_pkg;
+
+	pr_info("NPP %u MDPP %u\n", __amd_nodes_per_pkg, __max_die_per_package);
 }
author
#25 lore raw

Hi Thomas,

Quoted 26 lines · Thomas Gleixner

Laura!

arch/x86/kernel/cpu/topology_common.c void __init cpu_init_topology(struct cpu · line 220

Quoted 13 lines · Laura Nao
Quoted 1 line · Thomas Gleixner

Can you please replace that patch with the one below?

So, with this patch applied on top of ace278e7eca6 the kernel doesn't boot anymore - reference test job:
https://lava.collabora.dev/scheduler/job/13324010

I see the only change between the second and third patch you provided, besides the debug prints, is:

arch/x86/kernel/cpu/topology_common.c +0 -3
-	if (!topo_is_converted(c))
-		return;
-

Right. So this limits the area to search significantly.

Quoted 7 lines

Printing the debug information without this probably doesn't really help, but just in case it's useful: I tried excluding the change above from the patch while leaving everything else unchanged - reference test job:
https://lava.collabora.dev/scheduler/job/13324298 (also pasted the kernel log here for easier consultation:
https://pastebin.com/raw/TQBDvCah)

Quoted 7 lines
Quoted 1 line

Hope this helps,

It does. Good idea!

I just moved the exit check a bit so we should see the scan info. That should tell me what goes south.

Here's the full kernel log with the latest patch applied:
https://pastebin.com/raw/r2CkP396

Reference test job: https://lava.collabora.dev/scheduler/job/13328709

Thanks!

Laura

#26 lore raw

Laura!

Quoted 6 lines · Laura Nao
Quoted 2 lines · Thomas Gleixner

I just moved the exit check a bit so we should see the scan info. That should tell me what goes south.

Here's the full kernel log with the latest patch applied:
https://pastebin.com/raw/r2CkP396

Ok. Now I can see it.

	pr_info("NPP %u\n", tscan.amd_nodes_per_pkg);

The output is:

<6>[    0.000000] NPP 3

The original code then reads out 1, which is what's expected on that machine.

Now that bogus value '3' is used later in

    a %= b / 3

as b == 2, the division result is zero, which makes the %= operation crash with a division by zero. As there are no handlers yet, nothing to see.

The problem causing the bogus readout sits between my keyboard and my chair.

I'll send out the fixes with proper change logs later tonight.

Thanks for all your help!

tglx

#27 lore raw

cover x86/cpu/amd: Fixup the topology rework fallout

Testing at Collabora unearthed two issues in the new AMD topology parser code:

1) The CPUID 0x80000008 parser initializes the wrong topology domain level.

2) The NODEID_MSR parser uses bitfields in a union wrongly which results in reading out the wrong value and finally in a division by zero.

Many thanks to Laura for helping to debug this issue.

tglx

#28 lore raw

PATCH 1/2 x86/cpu/amd: Make the CPUID 0x80000008 parser correct

CPUID 0x80000008 ECX.cpu_nthreads describes the number of threads in the package. The parser uses this value to initialize the SMT domain level.

That's wrong because cpu_nthreads does not describe the number of threads per physical core. So this needs to set the CORE domain level and let the later parsers set the SMT shift if available.

Preset the SMT domain level with the assumption of one thread per core, which is correct ifrt here are no other CPUID leafs to parse, and propagate cpu_nthreads and the core level APIC bitwidth into the CORE domain.

Fixes f7fb3b2dd92c ("x86/cpu: Provide an AMD/HYGON specific topology parser")
---
 arch/x86/kernel/cpu/topology_amd.c |   24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/topology_amd.c +18 -6
@@ -29,11 +29,21 @@ static bool parse_8000_0008(struct topo_
 	if (!sft)
 		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
 	return true;
 }
 
-static void store_node(struct topo_scan *tscan, unsigned int nr_nodes, u16 node_id)
+static void store_node(struct topo_scan *tscan, u16 nr_nodes, u16 node_id)
 {
 	/*
 	 * Starting with Fam 17h the DIE domain could probably be used to
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_
 	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
 	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
 	 */
 	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
 		unsigned int nthreads = leaf.core_nthreads + 1;
 
 		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
#29 lore raw

[tip: x86/urgent] x86/cpu/amd: Make the CPUID 0x80000008 parser correct

The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     3a93fe9cdfa7bb45716aecf771be053a0ce2bdf0
Gitweb:        https://git.kernel.org/tip/3a93fe9cdfa7bb45716aecf771be053a0ce2bdf0
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Wed, 10 Apr 2024 21:45:27 +02:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 11 Apr 2024 14:18:29 +02:00

x86/cpu/amd: Make the CPUID 0x80000008 parser correct

CPUID 0x80000008 ECX.cpu_nthreads describes the number of threads in the package. The parser uses this value to initialize the SMT domain level.

That's wrong because cpu_nthreads does not describe the number of threads per physical core. So this needs to set the CORE domain level and let the later parsers set the SMT shift if available.

Preset the SMT domain level with the assumption of one thread per core, which is correct ifrt here are no other CPUID leafs to parse, and propagate cpu_nthreads and the core level APIC bitwidth into the CORE domain.

Fixes f7fb3b2dd92c ("x86/cpu: Provide an AMD/HYGON specific topology parser")
---
 arch/x86/kernel/cpu/topology_amd.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/topology_amd.c +18 -6
index 1a8b3ad..79a85a4 100644
@@ -29,11 +29,21 @@ static bool parse_8000_0008(struct topo_scan *tscan)
 	if (!sft)
 		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
 	return true;
 }
 
-static void store_node(struct topo_scan *tscan, unsigned int nr_nodes, u16 node_id)
+static void store_node(struct topo_scan *tscan, u16 nr_nodes, u16 node_id)
 {
 	/*
 	 * Starting with Fam 17h the DIE domain could probably be used to
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_0xb)
 	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
 	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
 	 */
 	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
 		unsigned int nthreads = leaf.core_nthreads + 1;
 
 		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
#30 lore raw

[tip: x86/urgent] x86/cpu/amd: Make the CPUID 0x80000008 parser correct

In reply to Thomas Gleixner
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     1b3108f6898ef2e03973d65255182792e94e2240
Gitweb:        https://git.kernel.org/tip/1b3108f6898ef2e03973d65255182792e94e2240
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Wed, 10 Apr 2024 21:45:27 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Fri, 12 Apr 2024 12:05:54 +02:00

x86/cpu/amd: Make the CPUID 0x80000008 parser correct

CPUID 0x80000008 ECX.cpu_nthreads describes the number of threads in the package. The parser uses this value to initialize the SMT domain level.

That's wrong because cpu_nthreads does not describe the number of threads per physical core. So this needs to set the CORE domain level and let the later parsers set the SMT shift if available.

Preset the SMT domain level with the assumption of one thread per core, which is correct ifrt here are no other CPUID leafs to parse, and propagate cpu_nthreads and the core level APIC bitwidth into the CORE domain.

Fixes f7fb3b2dd92c ("x86/cpu: Provide an AMD/HYGON specific topology parser")
---
 arch/x86/kernel/cpu/topology_amd.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/topology_amd.c +18 -6
index 1a8b3ad..79a85a4 100644
@@ -29,11 +29,21 @@ static bool parse_8000_0008(struct topo_scan *tscan)
 	if (!sft)
 		sft = get_count_order(ecx.cpu_nthreads + 1);
 
-	topology_set_dom(tscan, TOPO_SMT_DOMAIN, sft, ecx.cpu_nthreads + 1);
+	/*
+	 * cpu_nthreads describes the number of threads in the package
+	 * sft is the number of APIC ID bits per package
+	 *
+	 * As the number of actual threads per core is not described in
+	 * this leaf, just set the CORE domain shift and let the later
+	 * parsers set SMT shift. Assume one thread per core by default
+	 * which is correct if there are no other CPUID leafs to parse.
+	 */
+	topology_update_dom(tscan, TOPO_SMT_DOMAIN, 0, 1);
+	topology_set_dom(tscan, TOPO_CORE_DOMAIN, sft, ecx.cpu_nthreads + 1);
 	return true;
 }
 
-static void store_node(struct topo_scan *tscan, unsigned int nr_nodes, u16 node_id)
+static void store_node(struct topo_scan *tscan, u16 nr_nodes, u16 node_id)
 {
 	/*
 	 * Starting with Fam 17h the DIE domain could probably be used to
@@ -73,12 +83,14 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_0xb)
 	tscan->c->topo.initial_apicid = leaf.ext_apic_id;
 
 	/*
-	 * If leaf 0xb is available, then SMT shift is set already. If not
-	 * take it from ecx.threads_per_core and use topo_update_dom() -
-	 * topology_set_dom() would propagate and overwrite the already
-	 * propagated CORE level.
+	 * If leaf 0xb is available, then the domain shifts are set
+	 * already and nothing to do here.
 	 */
 	if (!has_0xb) {
+		/*
+		 * Leaf 0x80000008 set the CORE domain shift already.
+		 * Update the SMT domain, but do not propagate it.
+		 */
 		unsigned int nthreads = leaf.core_nthreads + 1;
 
 		topology_update_dom(tscan, TOPO_SMT_DOMAIN, get_count_order(nthreads), nthreads);
#31 lore raw

PATCH 2/2 x86/cpu/amd: Make the NODEID_MSR union actually work

In reply to Thomas Gleixner

A system with NODEID_MSR was reported to crash during early boot without any output.

The reason is that the union which is used for accessing the bitfields in the MSR is written wrongly and the resulting executable code accesses the wrong part of the MSR data.

As a consequence a later division by that value results in 0 and that result is used for another division as divisor, which obviously does not work well.

The magic world of C, unions and bitfields:

    union {
    	  u64   bita : 3,
	        bitb : 3;
	  u64   all;
    } x;

    x.all = foo();

    a = x.bita;
    b = x.bitb;

results in the effective executable code of:

   a = b = x.bita;

because bita and bitb are treated as union members and therefore both end up at bit offset 0.

Wrapping the bitfield into an anonymous struct:

    union {
    	  struct {
    	     u64  bita : 3,
	          bitb : 3;
          };
	  u64	  all;
    } x;

works like expected.

Rework the NODEID_MSR union in exactly that way to cure the problem.

Fixes f7fb3b2dd92c ("x86/cpu: Provide an AMD/HYGON specific topology parser")
---
 arch/x86/kernel/cpu/topology_amd.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/topology_amd.c +3 -3
@@ -121,13 +121,13 @@ static bool parse_8000_001e(struct topo_
 
 static bool parse_fam10h_node_id(struct topo_scan *tscan)
 {
-	struct {
-		union {
+	union {
+		struct {
 			u64	node_id		:  3,
 				nodes_per_pkg	:  3,
 				unused		: 58;
-			u64	msr;
 		};
+		u64		msr;
 	} nid;
 
 	if (!boot_cpu_has(X86_FEATURE_NODEID_MSR))
#32 lore raw

[tip: x86/urgent] x86/cpu/amd: Make the NODEID_MSR union actually work

The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     898e4996e76053bfa0f578629ec6b35baff4224b
Gitweb:        https://git.kernel.org/tip/898e4996e76053bfa0f578629ec6b35baff4224b
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Wed, 10 Apr 2024 21:45:28 +02:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Thu, 11 Apr 2024 14:18:29 +02:00

x86/cpu/amd: Make the NODEID_MSR union actually work

A system with NODEID_MSR was reported to crash during early boot without any output.

The reason is that the union which is used for accessing the bitfields in the MSR is written wrongly and the resulting executable code accesses the wrong part of the MSR data.

As a consequence a later division by that value results in 0 and that result is used for another division as divisor, which obviously does not work well.

The magic world of C, unions and bitfields:

    union {
    	  u64   bita : 3,
	        bitb : 3;
	  u64   all;
    } x;

    x.all = foo();

    a = x.bita;
    b = x.bitb;

results in the effective executable code of:

   a = b = x.bita;

because bita and bitb are treated as union members and therefore both end up at bit offset 0.

Wrapping the bitfield into an anonymous struct:

    union {
    	  struct {
    	     u64  bita : 3,
	          bitb : 3;
          };
	  u64	  all;
    } x;

works like expected.

Rework the NODEID_MSR union in exactly that way to cure the problem.

---
 arch/x86/kernel/cpu/topology_amd.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/topology_amd.c +3 -3
index 79a85a4..7f999ae 100644
@@ -121,13 +121,13 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_0xb)
 
 static bool parse_fam10h_node_id(struct topo_scan *tscan)
 {
-	struct {
-		union {
+	union {
+		struct {
 			u64	node_id		:  3,
 				nodes_per_pkg	:  3,
 				unused		: 58;
-			u64	msr;
 		};
+		u64		msr;
 	} nid;
 
 	if (!boot_cpu_has(X86_FEATURE_NODEID_MSR))
#33 lore raw

[tip: x86/urgent] x86/cpu/amd: Make the NODEID_MSR union actually work

In reply to Thomas Gleixner
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     c064b536a8f9ab7c8e204da8f5a22f7420d0b56c
Gitweb:        https://git.kernel.org/tip/c064b536a8f9ab7c8e204da8f5a22f7420d0b56c
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Wed, 10 Apr 2024 21:45:28 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Fri, 12 Apr 2024 12:05:54 +02:00

x86/cpu/amd: Make the NODEID_MSR union actually work

A system with NODEID_MSR was reported to crash during early boot without any output.

The reason is that the union which is used for accessing the bitfields in the MSR is written wrongly and the resulting executable code accesses the wrong part of the MSR data.

As a consequence a later division by that value results in 0 and that result is used for another division as divisor, which obviously does not work well.

The magic world of C, unions and bitfields:

    union {
    	  u64   bita : 3,
	        bitb : 3;
	  u64   all;
    } x;

    x.all = foo();

    a = x.bita;
    b = x.bitb;

results in the effective executable code of:

   a = b = x.bita;

because bita and bitb are treated as union members and therefore both end up at bit offset 0.

Wrapping the bitfield into an anonymous struct:

    union {
    	  struct {
    	     u64  bita : 3,
	          bitb : 3;
          };
	  u64	  all;
    } x;

works like expected.

Rework the NODEID_MSR union in exactly that way to cure the problem.

---
 arch/x86/kernel/cpu/topology_amd.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/topology_amd.c +3 -3
index 79a85a4..7f999ae 100644
@@ -121,13 +121,13 @@ static bool parse_8000_001e(struct topo_scan *tscan, bool has_0xb)
 
 static bool parse_fam10h_node_id(struct topo_scan *tscan)
 {
-	struct {
-		union {
+	union {
+		struct {
 			u64	node_id		:  3,
 				nodes_per_pkg	:  3,
 				unused		: 58;
-			u64	msr;
 		};
+		u64		msr;
 	} nid;
 
 	if (!boot_cpu_has(X86_FEATURE_NODEID_MSR))
author Tested-by on the v1 series
#34 lore raw

Re: [patch 0/2] x86/cpu/amd: Fixup the topology rework fallout

In reply to Thomas Gleixner

Hi Thomas,

Quoted 6 lines · Thomas Gleixner

Testing at Collabora unearthed two issues in the new AMD topology parser code:

1) The CPUID 0x80000008 parser initializes the wrong topology domain level.

Quoted 8 lines

2) The NODEID_MSR parser uses bitfields in a union wrongly which results in reading out the wrong value and finally in a division by zero.

Many thanks to Laura for helping to debug this issue.

tglx

Thanks a lot for investigating and solving the issue!

I confirm that with this series applied the kernel boots correctly on all three AMD Stoney Ridge Chromebooks that were affected by the regression.

I tested the patches on top of c749ce39 (culprit commit identified by the bisection) - reference test job:
https://lava.collabora.dev/scheduler/job/13339645

The series doesn't apply directly to next, but I manually applied the changes on top of next-20240411 and can confirm the kernel boots correctly with this revision too - reference test job:
https://lava.collabora.dev/scheduler/job/13340321

The regression was originally reported by KernelCI, so:

I'll make sure to update the Regzbot tag when the series is merged.

Best,

Laura

#35 lore raw

Re: [patch 0/2] x86/cpu/amd: Fixup the topology rework fallout

Quoted 17 lines · Laura Nao
Quoted 6 lines · Thomas Gleixner

Testing at Collabora unearthed two issues in the new AMD topology parser code:

1) The CPUID 0x80000008 parser initializes the wrong topology domain level.

Quoted 8 lines

2) The NODEID_MSR parser uses bitfields in a union wrongly which results in reading out the wrong value and finally in a division by zero.

Many thanks to Laura for helping to debug this issue.

tglx

Quoted 8 lines

Thanks a lot for investigating and solving the issue!> [...]

The regression was originally reported by KernelCI, so:

I'll make sure to update the Regzbot tag when the series is merged.

No need to wait, we can do that now:

#regzbot fix: x86/cpu/amd: Make the NODEID_MSR union actually work

But ideally Thomas would add Link: or Closes: tag to the patch description (e.g.

Closes:
https://lore.kernel.org/all/[email protected]/

) just like Linus asked him to do a while ago already[1], as then this would not be necessary at all. ;) (SCNR)

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

[1]
https://lore.kernel.org/all/CAHk-=wjMmSZzMJ3Xnskdg4+GGz=5p5p+GSYyFBTh0f-DgvdBWg@mail.gmail.com/

#36 lore raw

Re: [patch 0/2] x86/cpu/amd: Fixup the topology rework fallout

Quoted 5 lines · Linux regression tracking (Thorsten Leemhuis)

On 11.04.24 13:27, Laura Nao wrote:
No need to wait, we can do that now:

#regzbot fix: x86/cpu/amd: Make the NODEID_MSR union actually work

Quoted 8 lines

But ideally Thomas would add Link: or Closes: tag to the patch description (e.g.

Closes:
https://lore.kernel.org/all/[email protected]/

) just like Linus asked him to do a while ago already[1], as then this would not be necessary at all. ;) (SCNR)

Will do when applying them and I try to remember that Closes thing, but you know at my age ....

Thanks,

tglx