Skip to contentSkip to conversation

Fix community TSC ADJUST breakage causing TSC failure

PATCH In review Mike Travis 2 participants 11 messages Oct 2, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Fix community TSC ADJUST breakage causing TSC failure

Fix a breakage caused by enabling early tsc initialization which bypasses a check that disables the forcing of TSC ADJUST to 0 for chassis 0. This is common on systems where all the chassis start up asynchronously so which chassis should have a TSC ADJUST value of 0 is not predictable.

The solution is to add a check earlier than this early tsc init to disable the potential of it incorrectly adjusting TSC ADJUST values that are already correctly initialized.

* Patch 1 adds an early callable function (right after efi_init) that will check if this system might be a UV system.

* Patch 2 adds code into setup_arch() which disables the call to tsc_early_init. This function calls the TSC adjust initialization functions before UV setup has checked if BIOS has set the multi-chassis sync flags. Further references are in the patch.

#1 lore raw

PATCH 1/2 x86/platform/uv: Add is_early_uv_system check

Introduce is_early_uv_system() which uses efi.uv_systab to decide early in the boot process whether we're on a UV system.

This is needed to skip other early setup/init code that might break UV platform if done too early such as before necessary ACPI tables parsing takes place.

---
 arch/x86/include/asm/uv/uv.h |    6 ++++++
 1 file changed, 6 insertions(+)
arch/x86/include/asm/uv/uv.h +6 -0
@@ -10,8 +10,13 @@ struct cpumask;
 struct mm_struct;
 
 #ifdef CONFIG_X86_UV
+#include <linux/efi.h>
 
 extern enum uv_system_type get_uv_system_type(void);
+static inline int is_early_uv_system(void)
+{
+	return !((efi.uv_systab == EFI_INVALID_TABLE_ADDR) || !efi.uv_systab);
+}
 extern int is_uv_system(void);
 extern int is_uv_hubless(void);
 extern void uv_cpu_init(void);
@@ -23,6 +28,7 @@ extern const struct cpumask *uv_flush_tl
 #else	/* X86_UV */
 
 static inline enum uv_system_type get_uv_system_type(void) { return UV_NONE; }
+static inline int is_early_uv_system(void)	{ return 0; }
 static inline int is_uv_system(void)	{ return 0; }
 static inline int is_uv_hubless(void)	{ return 0; }
 static inline void uv_cpu_init(void)	{ }
#2 lore raw

Re: [PATCH 1/2] x86/platform/uv: Add is_early_uv_system check

Quoted 9 lines · Mike Travis

Introduce is_early_uv_system() which uses efi.uv_systab to decide early in the boot process whether we're on a UV system.

This is needed to skip other early setup/init code that might break UV platform if done too early such as before necessary ACPI tables parsing takes place.

This SOB chain is wrong. I don't see how Hedi is involved in this. If this wants to express that the patch was developed together please see Documentation for the Co-developed-by tag.

Quoted 9 lines
---
 arch/x86/include/asm/uv/uv.h |    6 ++++++
 1 file changed, 6 insertions(+)

arch/x86/include/asm/uv/uv.h struct cpumask; · line 16

Quoted 7 lines
arch/x86/include/asm/uv/uv.h +2 -0
@@ -10,8 +10,13 @@ struct cpumask;
 struct mm_struct;
 
 #ifdef CONFIG_X86_UV
+#include <linux/efi.h>
 
 extern enum uv_system_type get_uv_system_type(void);
+static inline int is_early_uv_system(void)

Can you please make this bool?

arch/x86/include/asm/uv/uv.h struct cpumask; · line 19

Quoted 3 lines
arch/x86/include/asm/uv/uv.h +3 -0
+{
+	return !((efi.uv_systab == EFI_INVALID_TABLE_ADDR) || !efi.uv_systab);
+}

Thanks,

tglx

#3 lore raw

Re: [PATCH 1/2] x86/platform/uv: Add is_early_uv_system check

Quoted 9 lines · Thomas Gleixner
Quoted 7 lines · Mike Travis

Introduce is_early_uv_system() which uses efi.uv_systab to decide early in the boot process whether we're on a UV system.

This is needed to skip other early setup/init code that might break UV platform if done too early such as before necessary ACPI tables parsing takes place.

Quoted 6 lines
Quoted 2 lines

This SOB chain is wrong. I don't see how Hedi is involved in this. If this wants to express that the patch was developed together please see Documentation for the Co-developed-by tag.

Hedi suggested a "Suggested-by:" tag, would that suffice?

Quoted 28 lines
Quoted 10 lines
---
  arch/x86/include/asm/uv/uv.h |    6 ++++++
  1 file changed, 6 insertions(+)

arch/x86/include/asm/uv/uv.h struct cpumask; · line 16

Quoted 7 lines
arch/x86/include/asm/uv/uv.h +2 -0
@@ -10,8 +10,13 @@ struct cpumask;
  struct mm_struct;
 
  #ifdef CONFIG_X86_UV
+#include <linux/efi.h>
 
  extern enum uv_system_type get_uv_system_type(void);
+static inline int is_early_uv_system(void)

Can you please make this bool?

arch/x86/include/asm/uv/uv.h struct cpumask; · line 19

Quoted 3 lines
arch/x86/include/asm/uv/uv.h +3 -0
+{
+	return !((efi.uv_systab == EFI_INVALID_TABLE_ADDR) || !efi.uv_systab);
+}

Thanks,

tglx

#4 lore raw

Re: [PATCH 1/2] x86/platform/uv: Add is_early_uv_system check

Quoted 10 lines · Mike Travis
Quoted 9 lines · Thomas Gleixner
Quoted 7 lines · Mike Travis

Introduce is_early_uv_system() which uses efi.uv_systab to decide early in the boot process whether we're on a UV system.

This is needed to skip other early setup/init code that might break UV platform if done too early such as before necessary ACPI tables parsing takes place.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

This SOB chain is wrong. I don't see how Hedi is involved in this. If this wants to express that the patch was developed together please see Documentation for the Co-developed-by tag.

Hedi suggested a "Suggested-by:" tag, would that suffice?

Sure, instead of the SOB.

Thanks,

tglx

#5 lore raw

PATCH 2/2 x86/tsc: Fix UV TSC initialization

In reply to Mike Travis

Fix regression introduced by

commit cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")

as it changed setup_arch() so that it now calls tsc_early_init() before acpi_boot_table_init() which is a necessary step, in the case of UV systems, to inform tsc_sanitize_first_cpu() that we're on a platform with async TSC resets as documented in

commit 341102c3ef29 ("x86/tsc: Add option that TSC on Socket 0 being non-zero is valid")

Fix by skipping tsc_early_init() on UV systems and let TSC initialization take place later in tsc_init().

Fixes cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")
---
 arch/x86/kernel/setup.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/setup.c +5 -1
@@ -117,6 +117,7 @@
 #include <asm/microcode.h>
 #include <asm/kaslr.h>
 #include <asm/unwind.h>
+#include <asm/uv/uv.h>
 
 /*
  * max_low_pfn_mapped: highest direct mapped pfn under 4GB
@@ -1015,7 +1016,10 @@ void __init setup_arch(char **cmdline_p)
 	 */
 	init_hypervisor_platform();
 
-	tsc_early_init();
+	/* UV TSC multi-chassis synchronization already set, don't change it */
+	if (!is_early_uv_system())
+		tsc_early_init();
+
 	x86_init.resources.probe_roms();
 
 	/* after parse_early_param, so could debug it */
#6 lore raw

Re: [PATCH 2/2] x86/tsc: Fix UV TSC initialization

Quoted 9 lines · Mike Travis

Fix regression introduced by

commit cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")

as it changed setup_arch() so that it now calls tsc_early_init() before acpi_boot_table_init() which is a necessary step, in the case of UV systems, to inform tsc_sanitize_first_cpu() that we're on a platform with async TSC resets as documented in

Quoted 8 lines

commit 341102c3ef29 ("x86/tsc: Add option that TSC on Socket 0 being non-zero is valid")

Fix by skipping tsc_early_init() on UV systems and let TSC initialization take place later in tsc_init().

Fixes cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")

See previous mail.

Quoted 17 lines
---
 arch/x86/kernel/setup.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/setup.c +1 -0
@@ -117,6 +117,7 @@
 #include <asm/microcode.h>
 #include <asm/kaslr.h>
 #include <asm/unwind.h>
+#include <asm/uv/uv.h>
 
 /*
  * max_low_pfn_mapped: highest direct mapped pfn under 4GB

arch/x86/kernel/setup.c void __init setup_arch(char **cmdline_p) · line 1021

Quoted 7 lines
arch/x86/kernel/setup.c +3 -1
@@ -1015,7 +1016,10 @@ void __init setup_arch(char **cmdline_p)
 	 */
 	init_hypervisor_platform();
 
-	tsc_early_init();
+	/* UV TSC multi-chassis synchronization already set, don't change it */
+	if (!is_early_uv_system())
+		tsc_early_init();

tsc_early_init() has already a check which makes it not invoke the calibration code, so please put your uv check into that.

Thanks,

tglx

#7 lore raw

Re: [PATCH 2/2] x86/tsc: Fix UV TSC initialization

Quoted 42 lines · Thomas Gleixner
Quoted 9 lines · Mike Travis

Fix regression introduced by

commit cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")

as it changed setup_arch() so that it now calls tsc_early_init() before acpi_boot_table_init() which is a necessary step, in the case of UV systems, to inform tsc_sanitize_first_cpu() that we're on a platform with async TSC resets as documented in

Quoted 8 lines

commit 341102c3ef29 ("x86/tsc: Add option that TSC on Socket 0 being non-zero is valid")

Fix by skipping tsc_early_init() on UV systems and let TSC initialization take place later in tsc_init().

Fixes cf7a63ef4e02 ("x86/tsc: Calibrate tsc only once")

See previous mail.

Quoted 20 lines
---
  arch/x86/kernel/setup.c |    6 +++++-
  1 file changed, 5 insertions(+), 1 deletion(-)
arch/x86/kernel/setup.c +1 -0
@@ -117,6 +117,7 @@
  #include <asm/microcode.h>
  #include <asm/kaslr.h>
  #include <asm/unwind.h>
+#include <asm/uv/uv.h>
 
  /*
   * max_low_pfn_mapped: highest direct mapped pfn under 4GB
@@ -1015,7 +1016,10 @@ void __init setup_arch(char **cmdline_p)
  	 */
  	init_hypervisor_platform();
 
Quoted 7 lines

arch/x86/kernel/setup.c void __init setup_arch(char **cmdline_p) · line 1021

Quoted 4 lines
arch/x86/kernel/setup.c +3 -1
-	tsc_early_init();
+	/* UV TSC multi-chassis synchronization already set, don't change it */
+	if (!is_early_uv_system())
+		tsc_early_init();

tsc_early_init() has already a check which makes it not invoke the calibration code, so please put your uv check into that.

Will do, thanks.

Quoted 5 lines

Thanks,

tglx

#9 lore raw
Quoted 4 lines · Thomas Gleixner

On Mon, 1 Oct 2018, Mike Travis wrote:

I'm not sure what a 'community TSC adjust breakage' means.

I probably should have cleaned that up. The title is from our internal bug report and we target where the earliest place where the fix should be applied. "Community" is shorthand for the Linux Community Kernel (as opposed to internal or distro specific problems).

#10 lore raw
Quoted 10 lines · Mike
Quoted 4 lines · Thomas Gleixner

On Mon, 1 Oct 2018, Mike Travis wrote:

I'm not sure what a 'community TSC adjust breakage' means.

I probably should have cleaned that up. The title is from our internal bug report and we target where the earliest place where the fix should be applied. "Community" is shorthand for the Linux Community Kernel (as opposed to internal or distro specific problems).

That was my guess. I just found the subject line amusing.

Thanks,

tglx