Skip to contentSkip to conversation

x86: Two fixes related to Xen PV guest mode

PATCH In review Juergen Gross 3 participants 11 messages Apr 5, 2024 – May 2, 2024

Subsystems (3)
lore
Share & tools
lore

Juergen Gross

author maintainer
#0 lore raw

cover x86: Two fixes related to Xen PV guest mode

These are 2 fixes for issues introduced by topology related changes added in the 6.9 merge window.

Juergen Gross (2):
x86/cpu: fix BSP detection when running as Xen PV guest x86/xen: return a sane initial apic id when running as PV guest

 arch/x86/kernel/cpu/topology.c |  2 +-
 arch/x86/xen/enlighten_pv.c    | 10 +++++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
Signature · 1 line
2.35.3
author maintainer
#1 lore raw

PATCH 1/2 x86/cpu: fix BSP detection when running as Xen PV guest

When booting as a Xen PV guest the boot processor isn't detected correctly and the following message is shown:

CPU topo: Boot CPU APIC ID not the first enumerated APIC ID: 0 > 1

Additionally this results in one CPU being ignored.

Fix that by calling the BSP detection logic when registering the boot CPU's APIC, too.

Fixes 5c5682b9f87a ("x86/cpu: Detect real BSP on crash kernels")
---
 arch/x86/kernel/cpu/topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/topology.c +1 -1
index aaca8d235dc2..23c3db5e6396 100644
@@ -255,7 +255,7 @@ void __init topology_register_boot_apic(u32 apic_id)
 	WARN_ON_ONCE(topo_info.boot_cpu_apic_id != BAD_APICID);
 
 	topo_info.boot_cpu_apic_id = apic_id;
-	topo_register_apic(apic_id, CPU_ACPIID_INVALID, true);
+	topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);
 }
 
 /**
Signature · 1 line
2.35.3
author maintainer
#2 lore raw

Re: [PATCH 1/2] x86/cpu: fix BSP detection when running as Xen PV guest

Ping?

Quoted 29 lines · Juergen Gross

When booting as a Xen PV guest the boot processor isn't detected correctly and the following message is shown:

CPU topo: Boot CPU APIC ID not the first enumerated APIC ID: 0 > 1

Additionally this results in one CPU being ignored.

Fix that by calling the BSP detection logic when registering the boot CPU's APIC, too.

Fixes 5c5682b9f87a ("x86/cpu: Detect real BSP on crash kernels")
---
  arch/x86/kernel/cpu/topology.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/topology.c +1 -1
index aaca8d235dc2..23c3db5e6396 100644
@@ -255,7 +255,7 @@ void __init topology_register_boot_apic(u32 apic_id)
  	WARN_ON_ONCE(topo_info.boot_cpu_apic_id != BAD_APICID);
 
  	topo_info.boot_cpu_apic_id = apic_id;
-	topo_register_apic(apic_id, CPU_ACPIID_INVALID, true);
+	topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);
  }
 
  /**
author maintainer
#3 lore raw

Re: [PATCH 1/2] x86/cpu: fix BSP detection when running as Xen PV guest

PING!!!!

This patch fixes a regression introduced in 6.9-rc1.

Please consider taking the patch in the 6.9 cycle!

#regzbot ^introduced: 5c5682b9f87a

Juergen

Quoted 31 lines · Juergen Gross

Ping?

Quoted 27 lines · Juergen Gross

When booting as a Xen PV guest the boot processor isn't detected correctly and the following message is shown:

   CPU topo: Boot CPU APIC ID not the first enumerated APIC ID: 0 > 1

Additionally this results in one CPU being ignored.

Fix that by calling the BSP detection logic when registering the boot CPU's APIC, too.

Fixes 5c5682b9f87a ("x86/cpu: Detect real BSP on crash kernels")

---
  arch/x86/kernel/cpu/topology.c | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/topology.c +1 -1
index aaca8d235dc2..23c3db5e6396 100644
@@ -255,7 +255,7 @@ void __init topology_register_boot_apic(u32 apic_id)
       WARN_ON_ONCE(topo_info.boot_cpu_apic_id != BAD_APICID);
       topo_info.boot_cpu_apic_id = apic_id;
-    topo_register_apic(apic_id, CPU_ACPIID_INVALID, true);
+    topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);
   }
   /**
#4 lore raw

Re: [PATCH 1/2] x86/cpu: fix BSP detection when running as Xen PV guest

In reply to Juergen Gross
Quoted 21 lines · Juergen Gross

When booting as a Xen PV guest the boot processor isn't detected correctly and the following message is shown:

CPU topo: Boot CPU APIC ID not the first enumerated APIC ID: 0 > 1

Additionally this results in one CPU being ignored.

Fix that by calling the BSP detection logic when registering the boot CPU's APIC, too.

Fixes 5c5682b9f87a ("x86/cpu: Detect real BSP on crash kernels")
---
 arch/x86/kernel/cpu/topology.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/topology.c +0 -0
index aaca8d235dc2..23c3db5e6396 100644

arch/x86/kernel/cpu/topology.c void __init topology_register_boot_apic(u32 apic_id) · line 258

Quoted 5 lines
arch/x86/kernel/cpu/topology.c +1 -1
@@ -255,7 +255,7 @@ void __init topology_register_boot_apic(u32 apic_id)
 	WARN_ON_ONCE(topo_info.boot_cpu_apic_id != BAD_APICID);
 
 	topo_info.boot_cpu_apic_id = apic_id;
-	topo_register_apic(apic_id, CPU_ACPIID_INVALID, true);
+	topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);

No. This does not fix anything at all. It just papers over the underlying problem.

Thanks,

tglx
---

arch/x86/xen/smp_pv.c +2 -2
index 27d1a5b7f571..ac41d83b38d3 100644
@@ -154,9 +154,9 @@ static void __init xen_pv_smp_config(void)
 	u32 apicid = 0;
 	int i;
 
-	topology_register_boot_apic(apicid++);
+	topology_register_boot_apic(apicid);
 
-	for (i = 1; i < nr_cpu_ids; i++)
+	for (i = 0; i < nr_cpu_ids; i++)
 		topology_register_apic(apicid++, CPU_ACPIID_INVALID, true);
 
 	/* Pretend to be a proper enumerated system */
author maintainer
#5 lore raw

Re: [PATCH 1/2] x86/cpu: fix BSP detection when running as Xen PV guest

Quoted 44 lines · Thomas Gleixner
Quoted 21 lines · Juergen Gross

When booting as a Xen PV guest the boot processor isn't detected correctly and the following message is shown:

CPU topo: Boot CPU APIC ID not the first enumerated APIC ID: 0 > 1

Additionally this results in one CPU being ignored.

Fix that by calling the BSP detection logic when registering the boot CPU's APIC, too.

Fixes 5c5682b9f87a ("x86/cpu: Detect real BSP on crash kernels")
---
  arch/x86/kernel/cpu/topology.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/topology.c +0 -0
index aaca8d235dc2..23c3db5e6396 100644

arch/x86/kernel/cpu/topology.c void __init topology_register_boot_apic(u32 apic_id) · line 258

Quoted 5 lines
arch/x86/kernel/cpu/topology.c +1 -1
@@ -255,7 +255,7 @@ void __init topology_register_boot_apic(u32 apic_id)
  	WARN_ON_ONCE(topo_info.boot_cpu_apic_id != BAD_APICID);
 
  	topo_info.boot_cpu_apic_id = apic_id;
-	topo_register_apic(apic_id, CPU_ACPIID_INVALID, true);
+	topology_register_apic(apic_id, CPU_ACPIID_INVALID, true);

No. This does not fix anything at all. It just papers over the underlying problem.

Thanks,

tglx
---

arch/x86/xen/smp_pv.c +0 -0
index 27d1a5b7f571..ac41d83b38d3 100644
@@ -154,9 +154,9 @@ static void __init xen_pv_smp_config(void)
  	u32 apicid = 0;

arch/x86/xen/smp_pv.c static void __init xen_pv_smp_config(void) · line 162

Quoted 10 lines
arch/x86/xen/smp_pv.c +2 -2
  	int i;
 
-	topology_register_boot_apic(apicid++);
+	topology_register_boot_apic(apicid);
 
-	for (i = 1; i < nr_cpu_ids; i++)
+	for (i = 0; i < nr_cpu_ids; i++)
  		topology_register_apic(apicid++, CPU_ACPIID_INVALID, true);
 
  	/* Pretend to be a proper enumerated system */

Thanks, works great.

Do you want it to send as your patch, or should I add your Signed-off-by or your Suggested-by?

Juergen

#6 lore raw

PATCH 1/1 x86/xen/smp_pv: Register the boot CPU APIC properly

The topology core expects the boot APIC to be registered from earhy APIC detection first and then again when the firmware tables are evaluated. This is used for detecting the real BSP CPU on a kexec kernel.

The recent conversion of XEN/PV to register fake APIC IDs failed to register the boot CPU APIC correctly as it only registers it once. This causes the BSP detection mechanism to trigger wrongly:

CPU topo: Boot CPU APIC ID not the first enumerated APIC ID: 0 > 1

Additionally this results in one CPU being ignored.

Register the boot CPU APIC twice so that the XEN/PV fake enumeration behaves like real firmware.

Fixes e75307023466 ("x86/xen/smp_pv: Register fake APICs")
---
 arch/x86/xen/smp_pv.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/xen/smp_pv.c +2 -2
@@ -154,9 +154,9 @@ static void __init xen_pv_smp_config(voi
 	u32 apicid = 0;
 	int i;
 
-	topology_register_boot_apic(apicid++);
+	topology_register_boot_apic(apicid);
 
-	for (i = 1; i < nr_cpu_ids; i++)
+	for (i = 0; i < nr_cpu_ids; i++)
 		topology_register_apic(apicid++, CPU_ACPIID_INVALID, true);
 
 	/* Pretend to be a proper enumerated system */
author maintainer Reviewed-by on v1 1/1
#7 lore raw

Re: [PATCH] x86/xen/smp_pv: Register the boot CPU APIC properly

Quoted 12 lines · Thomas Gleixner

The topology core expects the boot APIC to be registered from earhy APIC detection first and then again when the firmware tables are evaluated. This is used for detecting the real BSP CPU on a kexec kernel.

The recent conversion of XEN/PV to register fake APIC IDs failed to register the boot CPU APIC correctly as it only registers it once. This causes the BSP detection mechanism to trigger wrongly:

CPU topo: Boot CPU APIC ID not the first enumerated APIC ID: 0 > 1

Additionally this results in one CPU being ignored.

Quoted 7 lines

Register the boot CPU APIC twice so that the XEN/PV fake enumeration behaves like real firmware.

Fixes e75307023466 ("x86/xen/smp_pv: Register fake APICs")

Thanks for the patch, I'll take it via the Xen tree.

Juergen

Quoted 18 lines
---
  arch/x86/xen/smp_pv.c |    4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/xen/smp_pv.c +2 -2
@@ -154,9 +154,9 @@ static void __init xen_pv_smp_config(voi
  	u32 apicid = 0;
  	int i;
 
-	topology_register_boot_apic(apicid++);
+	topology_register_boot_apic(apicid);
 
-	for (i = 1; i < nr_cpu_ids; i++)
+	for (i = 0; i < nr_cpu_ids; i++)
  		topology_register_apic(apicid++, CPU_ACPIID_INVALID, true);
 
  	/* Pretend to be a proper enumerated system */
author maintainer
#8 lore raw

PATCH 2/2 x86/xen: return a sane initial apic id when running as PV guest

In reply to Juergen Gross

With recent sanity checks for topology information added, there are now warnings issued for APs when running as a Xen PV guest:

[Firmware Bug]: CPU 1: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0001

This is due to the initial APIC ID obtained via CPUID for PV guests is always 0.

Avoid the warnings by synthesizing the CPUID data to contain the same initial APIC ID as xen_pv_smp_config() is using for registering the APIC IDs of all CPUs.

Fixes 52128a7a21f7 ("86/cpu/topology: Make the APIC mismatch warnings complete")
---
 arch/x86/xen/enlighten_pv.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/xen/enlighten_pv.c +9 -1
index ace2eb054053..965e4ca36024 100644
@@ -219,13 +219,20 @@ static __read_mostly unsigned int cpuid_leaf5_edx_val;
 static void xen_cpuid(unsigned int *ax, unsigned int *bx,
 		      unsigned int *cx, unsigned int *dx)
 {
-	unsigned maskebx = ~0;
+	unsigned int maskebx = ~0;
+	unsigned int or_ebx = 0;
 
 	/*
 	 * Mask out inconvenient features, to try and disable as many
 	 * unsupported kernel subsystems as possible.
 	 */
 	switch (*ax) {
+	case 0x1:
+		/* Replace initial APIC ID in bits 24-31 of EBX. */
+		maskebx = 0x00ffffff;
+		or_ebx = smp_processor_id() << 24;
+		break;
+
 	case CPUID_MWAIT_LEAF:
 		/* Synthesize the values.. */
 		*ax = 0;
@@ -248,6 +255,7 @@ static void xen_cpuid(unsigned int *ax, unsigned int *bx,
 		: "0" (*ax), "2" (*cx));
 
 	*bx &= maskebx;
+	*bx |= or_ebx;
 }
 
 static bool __init xen_check_mwait(void)
Signature · 1 line
2.35.3
#9 lore raw

Re: [PATCH 2/2] x86/xen: return a sane initial apic id when running as PV guest

Quoted 7 lines · Juergen Gross

With recent sanity checks for topology information added, there are now warnings issued for APs when running as a Xen PV guest:

[Firmware Bug]: CPU 1: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0001

This is due to the initial APIC ID obtained via CPUID for PV guests is always 0.

/sigh

From Xen:

    switch ( leaf )
    {
    case 0x1:
        /* TODO: Rework topology logic. */
        res->b &= 0x00ffffffu;
        if ( is_hvm_domain(d) )
            res->b |= (v->vcpu_id * 2) << 24;

I think there's a very good chance it was random prior to Xen 4.6.  That used to come straight out of a CPUID value, so would get the APIC ID of whichever pCPU it was scheduled on.

Quoted 21 lines

Avoid the warnings by synthesizing the CPUID data to contain the same initial APIC ID as xen_pv_smp_config() is using for registering the APIC IDs of all CPUs.

Fixes 52128a7a21f7 ("86/cpu/topology: Make the APIC mismatch warnings complete")
---
 arch/x86/xen/enlighten_pv.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/xen/enlighten_pv.c +2 -1
index ace2eb054053..965e4ca36024 100644
@@ -219,13 +219,20 @@ static __read_mostly unsigned int cpuid_leaf5_edx_val;
 static void xen_cpuid(unsigned int *ax, unsigned int *bx,
 		      unsigned int *cx, unsigned int *dx)
 {
-	unsigned maskebx = ~0;
+	unsigned int maskebx = ~0;
+	unsigned int or_ebx = 0;

arch/x86/xen/enlighten_pv.c static __read_mostly unsigned int cpuid_leaf5_edx_val; · line 233

Quoted 10 lines
arch/x86/xen/enlighten_pv.c +4 -0
 
 	/*
 	 * Mask out inconvenient features, to try and disable as many
 	 * unsupported kernel subsystems as possible.
 	 */
 	switch (*ax) {
+	case 0x1:
+		/* Replace initial APIC ID in bits 24-31 of EBX. */
+		maskebx = 0x00ffffff;
+		or_ebx = smp_processor_id() << 24;

I think the comment wants to cross-reference explicitly with xen_pv_smp_config(), because what we care about here is the two sources of information matching.

Also while you're at it, the x2APIC ID in leaf 0xb.

~Andrew

author maintainer
#10 lore raw

Re: [PATCH 2/2] x86/xen: return a sane initial apic id when running as PV guest

Quoted 48 lines · Andrew Cooper
Quoted 7 lines · Juergen Gross

With recent sanity checks for topology information added, there are now warnings issued for APs when running as a Xen PV guest:

   [Firmware Bug]: CPU   1: APIC ID mismatch. CPUID: 0x0000 APIC: 0x0001

This is due to the initial APIC ID obtained via CPUID for PV guests is always 0.

/sigh

From Xen:

    switch ( leaf )
    {
    case 0x1:
        /* TODO: Rework topology logic. */
        res->b &= 0x00ffffffu;
        if ( is_hvm_domain(d) )
            res->b |= (v->vcpu_id * 2) << 24;

I think there's a very good chance it was random prior to Xen 4.6.  That used to come straight out of a CPUID value, so would get the APIC ID of whichever pCPU it was scheduled on.

Quoted 22 lines

Avoid the warnings by synthesizing the CPUID data to contain the same initial APIC ID as xen_pv_smp_config() is using for registering the APIC IDs of all CPUs.

Fixes 52128a7a21f7 ("86/cpu/topology: Make the APIC mismatch warnings complete")
---
  arch/x86/xen/enlighten_pv.c | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/xen/enlighten_pv.c +2 -1
index ace2eb054053..965e4ca36024 100644
@@ -219,13 +219,20 @@ static __read_mostly unsigned int cpuid_leaf5_edx_val;
  static void xen_cpuid(unsigned int *ax, unsigned int *bx,
  		      unsigned int *cx, unsigned int *dx)
  {
-	unsigned maskebx = ~0;
+	unsigned int maskebx = ~0;
+	unsigned int or_ebx = 0;
 
Quoted 13 lines

arch/x86/xen/enlighten_pv.c static __read_mostly unsigned int cpuid_leaf5_edx_val; · line 233

Quoted 9 lines
arch/x86/xen/enlighten_pv.c +4 -0
  	/*
  	 * Mask out inconvenient features, to try and disable as many
  	 * unsupported kernel subsystems as possible.
  	 */
  	switch (*ax) {
+	case 0x1:
+		/* Replace initial APIC ID in bits 24-31 of EBX. */
+		maskebx = 0x00ffffff;
+		or_ebx = smp_processor_id() << 24;

I think the comment wants to cross-reference explicitly with xen_pv_smp_config(), because what we care about here is the two sources of information matching.

I can add that as a comment. OTOH I'd really hope someone changing this code later would look into the commit message of the patch adding it. :-)

Quoted 2 lines

Also while you're at it, the x2APIC ID in leaf 0xb.

I'm not sure this is functionally relevant in PV guests.

Note that my patch is only meant to silence warnings during boot. It is not needed for the system working correctly (at least I think so).

Juergen