Skip to contentSkip to conversation

lib/s390x: fix SMP setup bug

PATCH In review Claudio Imbrenda 3 participants 6 messages Aug 18, 2022 – Aug 23, 2022 KVM

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lib/s390x: fix SMP setup bug

The lowcore pointer pointing to the current CPU (THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.

The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.

Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.

Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")
---
 lib/s390x/io.c  | 9 +++++++++
 lib/s390x/smp.c | 1 +
 2 files changed, 10 insertions(+)
lib/s390x/io.c +9 -0
index a4f1b113..fb7b7dda 100644
@@ -33,6 +33,15 @@ void puts(const char *s)
 
 void setup(void)
 {
+	struct cpu this_cpu_tmp = { 0 };
+
+	/*
+	 * Set a temporary empty struct cpu for the boot CPU, needed for
+	 * correct interrupt handling in the setup process.
+	 * smp_setup will allocate and set the permanent one.
+	 */
+	THIS_CPU = &this_cpu_tmp;
+
 	setup_args_progname(ipl_args);
 	setup_facilities();
 	sclp_read_info();
lib/s390x/smp.c +1 -0
index 0d98c17d..03d6d2a4 100644
@@ -353,6 +353,7 @@ void smp_setup(void)
 			cpus[0].stack = stackptr;
 			cpus[0].lowcore = (void *)0;
 			cpus[0].active = true;
+			THIS_CPU = &cpus[0];
 		}
 	}
 	spin_unlock(&lock);
Signature · 1 line
2.37.2
#1 lore raw

Quoting Claudio Imbrenda (2022-08-18 17:21:14)

Quoted 11 lines

The lowcore pointer pointing to the current CPU (THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.

The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.

Quoted 6 lines

Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.

Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")
Reviewed-and-tested-by Nico Boehr <[email protected]>
#2 lore raw
In reply to Claudio Imbrenda
Quoted 11 lines · Claudio Imbrenda

The lowcore pointer pointing to the current CPU (THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.

The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.

Quoted 6 lines

Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.

Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")

I've considered letting the IPL cpu have a static struct cpu and setting it up in cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.

Quoted 10 lines
---
  lib/s390x/io.c  | 9 +++++++++
  lib/s390x/smp.c | 1 +
  2 files changed, 10 insertions(+)
lib/s390x/io.c +0 -0
index a4f1b113..fb7b7dda 100644

lib/s390x/io.c void puts(const char *s) · line 36

Quoted 4 lines
lib/s390x/io.c +1 -0
@@ -33,6 +33,15 @@ void puts(const char *s)
 
  void setup(void)
  {
+	struct cpu this_cpu_tmp = { 0 };

We can setup these struct members here and memcpy in smp_setup() .addr = stap();
.stack = stackptr;
.lowcore = (void *)0;
.active = true;

Quoted 16 lines
lib/s390x/io.c +8 -0
+
+	/*
+	 * Set a temporary empty struct cpu for the boot CPU, needed for
+	 * correct interrupt handling in the setup process.
+	 * smp_setup will allocate and set the permanent one.
+	 */
+	THIS_CPU = &this_cpu_tmp;
+
  	setup_args_progname(ipl_args);
  	setup_facilities();
  	sclp_read_info();
lib/s390x/smp.c +0 -0
index 0d98c17d..03d6d2a4 100644

lib/s390x/smp.c void smp_setup(void) · line 356

Quoted 4 lines
lib/s390x/smp.c +1 -0
@@ -353,6 +353,7 @@ void smp_setup(void)
  			cpus[0].stack = stackptr;
  			cpus[0].lowcore = (void *)0;
  			cpus[0].active = true;
+			THIS_CPU = &cpus[0];

/* Override temporary struct cpu address with permanent one */

Quoted 3 lines
lib/s390x/smp.c +0 -0
  		}
  	}
  	spin_unlock(&lock);
#3 lore raw
Quoted 36 lines · On Fri, 19 Aug 2022 10:52:40 +0200 Janosch Frank <[email protected]> wrote:
Quoted 11 lines · Claudio Imbrenda

The lowcore pointer pointing to the current CPU (THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.

The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.

Quoted 6 lines

Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.

Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")

I've considered letting the IPL cpu have a static struct cpu and setting it up in cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.

Quoted 11 lines
---
  lib/s390x/io.c  | 9 +++++++++
  lib/s390x/smp.c | 1 +
  2 files changed, 10 insertions(+)
lib/s390x/io.c +0 -0
index a4f1b113..fb7b7dda 100644
@@ -33,6 +33,15 @@ void puts(const char *s)
 
Quoted 7 lines

lib/s390x/io.c void puts(const char *s) · line 36

Quoted 3 lines
lib/s390x/io.c +1 -0
  void setup(void)
  {
+	struct cpu this_cpu_tmp = { 0 };  

We can setup these struct members here and memcpy in smp_setup() .addr = stap();
.stack = stackptr;

stackptr is accessible in io.c (I would need to add an extern declaration)

Quoted 2 lines

.lowcore = (void *)0;
.active = true;

this temporary struct is then not accessible from smp_setup, so it can't be memcpy-ed.

if you really want something meaningful in the temporary struct, it has to be initialized in smp.c and called in io.c (something like smp_boot_cpu_tmp_setup(&this_cpu_tmp) ), but then still no memcpy.

in the end the struct cpu is needed only to allow interrupts to happen without crashes, I don't think we strictly need initialization

Quoted 16 lines
Quoted 16 lines
lib/s390x/io.c +8 -0
+
+	/*
+	 * Set a temporary empty struct cpu for the boot CPU, needed for
+	 * correct interrupt handling in the setup process.
+	 * smp_setup will allocate and set the permanent one.
+	 */
+	THIS_CPU = &this_cpu_tmp;
+
  	setup_args_progname(ipl_args);
  	setup_facilities();
  	sclp_read_info();
lib/s390x/smp.c +0 -1
index 0d98c17d..03d6d2a4 100644
--- a/lib/s390x/smp.c
Quoted 8 lines

lib/s390x/smp.c void smp_setup(void) · line 356

Quoted 6 lines

+++ b/lib/s390x/smp.c

lib/s390x/smp.c +1 -0
@@ -353,6 +353,7 @@ void smp_setup(void)
  			cpus[0].stack = stackptr;
  			cpus[0].lowcore = (void *)0;
  			cpus[0].active = true;
+			THIS_CPU = &cpus[0];  

/* Override temporary struct cpu address with permanent one */

will be done

Quoted 5 lines
Quoted 4 lines
lib/s390x/smp.c +0 -0
  		}
  	}
  	spin_unlock(&lock);  
#4 lore raw
Quoted 56 lines · Claudio Imbrenda
Quoted 36 lines · On Fri, 19 Aug 2022 10:52:40 +0200 Janosch Frank <[email protected]> wrote:
Quoted 11 lines · Claudio Imbrenda

The lowcore pointer pointing to the current CPU (THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.

The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.

Quoted 6 lines

Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.

Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")

I've considered letting the IPL cpu have a static struct cpu and setting it up in cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.

Quoted 11 lines
---
   lib/s390x/io.c  | 9 +++++++++
   lib/s390x/smp.c | 1 +
   2 files changed, 10 insertions(+)
lib/s390x/io.c +0 -0
index a4f1b113..fb7b7dda 100644
@@ -33,6 +33,15 @@ void puts(const char *s)
 
Quoted 7 lines

lib/s390x/io.c void puts(const char *s) · line 36

Quoted 3 lines
lib/s390x/io.c +1 -0
   void setup(void)
   {
+	struct cpu this_cpu_tmp = { 0 };

We can setup these struct members here and memcpy in smp_setup() .addr = stap();
.stack = stackptr;

stackptr is accessible in io.c (I would need to add an extern declaration)

Quoted 2 lines

.lowcore = (void *)0;
.active = true;

this temporary struct is then not accessible from smp_setup, so it can't be memcpy-ed.

Quoted 6 lines

if you really want something meaningful in the temporary struct, it has to be initialized in smp.c and called in io.c (something like smp_boot_cpu_tmp_setup(&this_cpu_tmp) ), but then still no memcpy.

in the end the struct cpu is needed only to allow interrupts to happen without crashes, I don't think we strictly need initialization

Ugh, this feels like a quick fix.
But alright, I've just tried setting it up from cstart64.S and it's way more ugly code so let's stick with this for now.

Quoted 34 lines
Quoted 17 lines
Quoted 17 lines
lib/s390x/io.c +8 -0
+
+	/*
+	 * Set a temporary empty struct cpu for the boot CPU, needed for
+	 * correct interrupt handling in the setup process.
+	 * smp_setup will allocate and set the permanent one.
+	 */
+	THIS_CPU = &this_cpu_tmp;
+
   	setup_args_progname(ipl_args);
   	setup_facilities();
   	sclp_read_info();
lib/s390x/smp.c +0 -1
index 0d98c17d..03d6d2a4 100644
--- a/lib/s390x/smp.c
Quoted 8 lines

lib/s390x/smp.c void smp_setup(void) · line 356

Quoted 6 lines

+++ b/lib/s390x/smp.c

lib/s390x/smp.c +1 -0
@@ -353,6 +353,7 @@ void smp_setup(void)
   			cpus[0].stack = stackptr;
   			cpus[0].lowcore = (void *)0;
   			cpus[0].active = true;
+			THIS_CPU = &cpus[0];

/* Override temporary struct cpu address with permanent one */

will be done

Quoted 5 lines
Quoted 4 lines
lib/s390x/smp.c +0 -0
   		}
   	}
   	spin_unlock(&lock);
Reviewed-by
#5 lore raw
Quoted 29 lines · Janosch Frank
Quoted 28 lines · Claudio Imbrenda
Quoted 25 lines · On Fri, 19 Aug 2022 10:52:40 +0200 Janosch Frank <[email protected]> wrote:
Quoted 11 lines · Claudio Imbrenda

The lowcore pointer pointing to the current CPU (THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.

The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.

Quoted 6 lines

Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.

Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")

I've considered letting the IPL cpu have a static struct cpu and setting it up in cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.

[...]

Quoted 8 lines
Quoted 8 lines

this temporary struct is then not accessible from smp_setup, so it can't be memcpy-ed.

if you really want something meaningful in the temporary struct, it has to be initialized in smp.c and called in io.c (something like smp_boot_cpu_tmp_setup(&this_cpu_tmp) ), but then still no memcpy.

Quoted 7 lines
Quoted 2 lines

in the end the struct cpu is needed only to allow interrupts to happen without crashes, I don't think we strictly need initialization

Ugh, this feels like a quick fix.
But alright, I've just tried setting it up from cstart64.S and it's way more ugly code so let's stick with this for now.

Anyway:

Thanks, picked